I recently launched my first ecommerce site, but Google Search Console—and some AI-based tools—cannot read my sitemap.xml file. The site uses strict firewall rules, although robots.txt and sitemap.xml are intended to remain publicly accessible. I can retrieve both files successfully with curl, and the frontend is served from a static Apache host. Cloudflare is not serving the files, and caching is bypassed for these paths. Other routes are restricted to visitors from Europe. What should I check to determine why Google cannot access the sitemap?
3 Answers
Search Console is useful for confirming the failure details, but the fact that a couple of pages are already indexed does not guarantee that the sitemap endpoint is accessible. Treat the sitemap as a separate public resource and test it independently from the same regions and networks that Google may use.
Start with the actual HTTP status code and response headers that Google receives. A sitemap can work in a browser or with curl while still being rejected by a firewall, redirect, authentication rule, malformed content type, or server error. Search Console’s crawl information and your Apache logs should help identify whether Google reached the host and what response it received.
The most likely issue is that your geo-based firewall is blocking Googlebot. A curl test from Europe only proves that the file works from Europe; Google’s crawler may be connecting from another region. Check the Apache access and error logs for Googlebot requests returning 403 or another failure. Also test the sitemap from a US-based connection using a Googlebot user agent. Make sure the firewall explicitly allows /sitemap.xml and /robots.txt, and verify that the response is a successful 200 with Content-Type: application/xml and no X-Robots-Tag: noindex.
Thanks, I’ll inspect the Apache logs and test from a US connection. I thought those two paths were exempt from the firewall, but I’ll verify that the exception applies all the way to the origin server.
That makes sense. I’ll compare the headers and status code from different locations and spend more time checking where the request is being blocked.