GPTBot, OAI-SearchBot and ChatGPT-User: Reading AI Bot Names in Your Server Logs
How to read AI crawler names in server logs, why spoofed user agents distort the picture, and which bots to separate before you change robots.txt.
Open a raw access log on a busy site and the AI traffic looks like noise: long user-agent strings, thousands of lines, a few names you half recognize. Teams usually respond in one of two ways. They block everything with "bot" in it, or they ignore the log entirely. Both are mistakes, and both come from not knowing which names mean what.
This post is a field guide to reading those names, and to the one trap that makes naive log reading unreliable.
Use the real names
The AI crawlers you will meet most often include GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI; ClaudeBot, Claude-User and Claude-SearchBot from Anthropic; PerplexityBot and Perplexity-User; Google-Agent and GoogleOther; Meta-WebIndexer, Meta-ExternalAgent and Meta-ExternalFetcher; and MistralAI-User.
One name you will not find is "ChatGPTBot". It does not exist. If a blog post or a plugin tells you to allow or block it, treat everything else it says with suspicion. Grepping your logs for it returns zero lines and tells you nothing.
Notice the pattern in the list. Vendors ship several agents, not one. Some names are crawlers that discover and index pages. Others, the ones ending in "-User", appear to act on behalf of a person asking a question at that moment. That difference matters for interpretation, because a fetch triggered by a live question is a closer sign of demand than a background crawl.
Why you should not lump them together
Treat each name as its own signal. A page fetched by a search-oriented bot has a different significance from one fetched by a user-triggered agent, and a single "AI traffic" line chart blends them.
A practical consequence: when you make a robots.txt decision, make it per agent. A blanket rule that blocks a whole vendor family can remove you from answers you wanted to be in. The reverse is also true. Allowing everything because "AI is good for us" ignores that you may have real reasons to restrict some agents.
The robots.txt side of the decision is yours and depends on your business. What the log gives you is evidence for it: which agents actually visit, how often, and which pages they want.
The spoofing trap
Anyone can put "GPTBot" in a user-agent header. Scrapers do. If you count log lines by user-agent string, some of your "OpenAI crawler" traffic is not from OpenAI at all, and your charts overstate AI interest in your site.
The reliable fix is to verify the source IP against the published ranges of the vendor, and discard lines that claim a name but come from somewhere else. Doing that by hand means collecting current IP lists for each vendor, keeping them updated, and joining them to every log line. It is possible and it is tedious, which is why it is usually skipped.
Promptwatch does this verification in Agent Analytics: its crawler logs are real-time and IP-verified, with spoofed user agents filtered out. If you have ever compared a raw log count with a vendor dashboard and wondered why they disagree, spoofing is a leading suspect.
What to look for once the names are clean
With verified lines per agent, four questions become answerable.
Which agents visit at all? Some sites discover that one vendor never fetches them, which usually points to a blocking rule or a CDN challenge.
Which pages do they request? Compare against your priority list. If bots keep hitting archive pages and never your pricing or product pages, internal linking and sitemap hygiene are suspects.
What status codes do they get? A page returning 403 or 5xx to a bot cannot be cited. Error rates per agent are among the quickest wins, because the fix is usually a single rule.
Does a fetch lead to a citation? This is the one raw logs cannot show on their own, since citations happen in the answer, not on your server. You need the prompt side joined to the log side.
Promptwatch's crawl-to-citation path does that join and reports a citation rate per page, with error tracking that explains failures. The path filters stack, and everything exports to CSV if you want to work in a spreadsheet anyway.
Getting logs in
Logs have to come from somewhere. Promptwatch lists several routes: Cloudflare through Logpush or an auto-deployed Worker, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint, or the WordPress plugin. If you already run Cloudflare, that is typically the shortest path. If you are on a platform that does not expose logs, the WordPress plugin or custom HTTP route may cover you.
Check the plan before you plan the project. Crawler logs are included from the Professional plan ($245 a month) up, with 25M logs on Professional and 100M on Business. Essential has none. Agency plans all include crawler logs, from 10M on Kick-off.
A short routine
- Connect logs and let a week accumulate.
- List the agents that visited, with verified IPs only.
- Sort by errors and fix the top three causes.
- Compare fetched pages against your priority list and note the gaps.
- Only then touch robots.txt, and do it per agent.
Step five is last on purpose. Most robots.txt edits made without log evidence are guesses, and guesses about crawler access are expensive because their effect is slow to show.
The recommendation
You do not need to become a log analyst. You need clean agent names, verified sources, and a join to citations. If you want that without maintaining IP lists, look at Agent Analytics in Promptwatch and connect your CDN first. The rest of your GEO work will be easier once you can see who is actually knocking.