Some requests to my site claim to be ChatGPT or GPTBot. Is the user agent enough to identify them, and what should I check before treating them differently from other bots?
How can I tell if a ChatGPT crawler is genuine?
Asked by
81b.sync
Answers
The user agent alone tells you what the request claims to be. Anyone can set that text, so it is not proof that a request came from OpenAI.
Check the source IP against the crawler IP ranges OpenAI publishes for the relevant bot. Keep those ranges updated, and distinguish a verified match from an unverified claim in your logs. An address outside a published range is worth treating cautiously, though it is not necessarily proof of malicious intent.
Reply to this answer
If you use Cloudflare, look at its Verified Bots feature rather than making your own allow rule based on the user agent. Check what that verification covers before relying on it, and keep an eye on requests that only claim to be a known crawler.
Reply to this answer
I would not give a request a free pass through security controls just because it says GPTBot. Some spoofed requests may be scraping, while others may probe for files such as .env or cloud credentials.
Make sure sensitive files are not publicly accessible in the first place. Then use verification and your normal rate limiting or blocking rules to decide how to handle suspicious traffic.
Reply to this answer
Add your answer
Answers are reviewed before publication.
81b.sync:
So a GPTBot label in analytics could include traffic that was never verified?
thimble-sized-ambition:
Yes. If the label is based only on the user agent, it includes anyone using that string.