I'm working with several websites that list companies alphabetically, with roughly 40–50 links under each letter. I want to filter the companies by geographical region without opening every company page individually. The location usually appears on the page behind each link rather than on the main directory page. Is there an efficient way to collect the links and search those pages for cities, countries, or regions?
4 Answers
If the location is already visible in the directory text, you can use your browser’s find feature to search for a city or region. Since it sounds like the location is usually on the linked pages instead, extract the links first with the browser console. For example: [...document.querySelectorAll('a')].map(a => a.href).join('n'). You can save the results and search or fetch the pages in batches. Be aware that browser requests to other domains may be blocked by CORS, so downloading them with a command-line tool or small script is often easier.
The directory may contain useful location clues in the link title or award name, such as a country appearing after a dash. Check the page HTML and link text before crawling every destination. If that information isn’t present, a script or data-extraction tool will save a lot of time compared with opening each page by hand.
For a list of hundreds or thousands of companies, automate the first stage: collect each company name and URL, download the pages with a script, then search the page text for a controlled list of countries, states, cities, or regions. You can also use an AI tool to classify the extracted text, but manually verify the results because locations may be written inconsistently or refer to an office, award category, or service area rather than headquarters.
A site-specific search may find many of them without any scripting. Try searches such as site:example.com "Manchester" or site:example.com "Spain", replacing the domain and location with what you need. This depends on the pages being indexed and won’t catch every variation, but it’s a quick first pass.
The location isn’t shown on the directory page, so the browser search doesn’t help much. I’ll try extracting the links and processing the linked pages instead.