I launched my first ecommerce website, but Google's tools cannot read or process my sitemap.xml file. I use fairly strict firewall rules, although robots.txt and sitemap.xml are supposed to remain accessible. The files load correctly when I test them with curl, and the frontend is hosted as static files on Apache. Cloudflare is not serving or managing the files, and caching is bypassed for both paths. Other routes are restricted to visitors from Europe. What should I check to determine why Google cannot fetch the sitemap?
3 Answers
The geographic firewall restriction is the most likely cause. Googlebot may crawl from an IP outside Europe, so a test from a European connection can succeed while Google receives a 403 or another blocked response. Check the Apache access and error logs for Googlebot requests, then test the sitemap from a US-based connection using a Googlebot user agent. Make sure both /sitemap.xml and /robots.txt bypass every firewall rule, including rules at the origin server. Also verify that the response is HTTP 200, uses Content-Type: application/xml, and does not include an X-Robots-Tag: noindex.
Start with the actual response Google receives rather than relying only on a local curl test. Check the status code, redirects, response headers, and body from several regions. Strict network rules can still block Googlebot even when the file appears publicly accessible from your own location. The crawl statistics and server logs should help identify whether the request reaches Apache and what response it gets.
Thanks. I’ll compare the headers and status codes from different locations and spend more time checking where the request is being stopped.
Confirm that the sitemap has been submitted in Google Search Console and inspect the reported fetch error. The fact that a couple of pages are already indexed does not prove that the sitemap is reachable; Google can discover pages through links or other signals independently. Pay particular attention to the firewall and origin server, since an edge-cache bypass alone will not help if Apache or another upstream rule blocks the crawler.
That makes sense. I’ll inspect the Apache logs and test from a US-based connection, even though those two paths are intended to be exempt from the firewall.