How can I find companies by location across a large directory of links?
Asked by
MellowPine47
I'm working with several websites that list companies alphabetically, with roughly 40โ50 links under each letter. I want to filter the companies by geographical region without opening every company page individually. The location usually appears on the page behind each link rather than on the main directory page. Is there an efficient way to collect the links and search those pages for cities, countries, or regions?
Answers
A site-specific search may find many of them without any scripting. Try searches such as site:example.com "Manchester" or site:example.com "Spain", replacing the domain and location with what you need. This depends on the pages being indexed and wonโt catch every variation, but itโs a quick first pass.
Reply to this answer
For a list of hundreds or thousands of companies, automate the first stage: collect each company name and URL, download the pages with a script, then search the page text for a controlled list of countries, states, cities, or regions. You can also use an AI tool to classify the extracted text, but manually verify the results because locations may be written inconsistently or refer to an office, award category, or service area rather than headquarters.
Reply to this answer
The directory may contain useful location clues in the link title or award name, such as a country appearing after a dash. Check the page HTML and link text before crawling every destination. If that information isnโt present, a script or data-extraction tool will save a lot of time compared with opening each page by hand.
Reply to this answer
If the location is already visible in the directory text, you can use your browserโs find feature to search for a city or region. Since it sounds like the location is usually on the linked pages instead, extract the links first with the browser console. For example: [...document.querySelectorAll('a')].map(a => a.href).join('n'). You can save the results and search or fetch the pages in batches. Be aware that browser requests to other domains may be blocked by CORS, so downloading them with a command-line tool or small script is often easier.
Reply to this answer
Add your answer
Answers are reviewed before publication.
MellowPine47: The location isnโt shown on the directory page, so the browser search doesnโt help much. Iโll try extracting the links and processing the linked pages instead.