How can I find companies by location across a large directory of links?

September 13, 2026 4 answers
Asked by MellowPine47
I'm working with several websites that list companies alphabetically, with roughly 40โ€“50 links under each letter. I want to filter the companies by geographical region without opening every company page individually. The location usually appears on the page behind each link rather than on the main directory page. Is there an efficient way to collect the links and search those pages for cities, countries, or regions?

Answers

MapleCircuit6
A site-specific search may find many of them without any scripting. Try searches such as site:example.com "Manchester" or site:example.com "Spain", replacing the domain and location with what you need. This depends on the pages being indexed and wonโ€™t catch every variation, but itโ€™s a quick first pass.
Reply to this answer
QuietOrbit29
For a list of hundreds or thousands of companies, automate the first stage: collect each company name and URL, download the pages with a script, then search the page text for a controlled list of countries, states, cities, or regions. You can also use an AI tool to classify the extracted text, but manually verify the results because locations may be written inconsistently or refer to an office, award category, or service area rather than headquarters.
Reply to this answer
SilverKite51
The directory may contain useful location clues in the link title or award name, such as a country appearing after a dash. Check the page HTML and link text before crawling every destination. If that information isnโ€™t present, a script or data-extraction tool will save a lot of time compared with opening each page by hand.
Reply to this answer
BrightCedar82
If the location is already visible in the directory text, you can use your browserโ€™s find feature to search for a city or region. Since it sounds like the location is usually on the linked pages instead, extract the links first with the browser console. For example: [...document.querySelectorAll('a')].map(a => a.href).join('n'). You can save the results and search or fetch the pages in batches. Be aware that browser requests to other domains may be blocked by CORS, so downloading them with a command-line tool or small script is often easier.

MellowPine47: The location isnโ€™t shown on the directory page, so the browser search doesnโ€™t help much. Iโ€™ll try extracting the links and processing the linked pages instead.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions