The AI Crawler Landscape in 2026: Who Is Crawling Your Site?
A practical guide to AI training bots, search crawlers, and user-triggered fetchers, plus the logs and checks needed to identify them.
An "AI crawler" is not one kind of visitor. The same provider may operate one bot for model training, another for a search index, and a user-triggered fetcher that retrieves a page while an answer is being composed. If those requests are grouped under one dashboard label, a marketing team can draw the wrong conclusion from perfectly valid logs.
The practical job is to identify the agent, understand its stated purpose, verify that the request is genuine where possible, and inspect what your server returned. Only then can you connect crawling with citations or human visits.
The three jobs hidden inside one label
Training crawlers collect material that may be used to develop future models. OpenAI's GPTBot and Anthropic's ClaudeBot fit this broad category. A publisher may have policy or licensing reasons to allow or block training collection. That decision should be made separately from search visibility.
Search index crawlers gather pages for retrieval in answer products. OAI-SearchBot, Claude-SearchBot, and PerplexityBot are examples listed in Promptwatch's crawlability documentation. Blocking a training bot does not necessarily remove a site from live search, while blocking a provider's search crawler can prevent its index from discovering the page.
User-triggered fetchers request content when a person asks a question or opens a link. ChatGPT-User, Claude-User, and Perplexity-User belong in this group. These requests are closer to an active answer event than a broad training crawl, but a request still does not prove that the page received a visible citation.
Google needs a separate note. Google-Extended is a robots control token, not a distinct crawler that appears in logs. Google's ordinary crawlers may fetch the page, while the token controls certain uses in Gemini systems. Treating Google-Extended as if it were a user agent request will produce a monitoring rule that cannot work.
Who may appear in your logs
OpenAI, Anthropic, Perplexity, Google, Meta, Mistral, Apple, Amazon, Microsoft, and other companies publish or use identifiable agents. Promptwatch's crawler directory is a convenient starting list, but user-agent directories age quickly. The page was originally published in September 2025 and updated in July 2026, according to its own publication record. Confirm any control rule against the provider's current documentation before deploying it.
Names alone are not enough. A request header can claim to be GPTBot without coming from OpenAI. For providers that publish network ranges, compare the source address with those ranges. Promptwatch's crawler insights guide says its product performs IP checks for supported providers and excludes requests that fail verification. Where a provider does not publish a verifiable range, label the traffic as user-agent identified rather than verified.
That distinction matters in reporting. A spike in unverified bot strings may be a scraper imitating a known agent. It should not be presented to management as sudden interest from an AI platform.
What a useful crawl record contains
Keep the timestamp, provider, exact user agent, source address or verification result, request method, path, query string, referrer, and HTTP status. The status tells you what the crawler received at the transport level. The path tells you whether it reached a money page, an expired URL, or a faceted duplicate.
Successful responses deserve another check: was the useful content present in the returned HTML? A browser can show content that was inserted later by JavaScript, while a crawler may only receive a sparse shell. A successful request is therefore proof of delivery, not proof that the page's argument, product data, or answer text was visible.
Look for patterns by agent and directory. Repeated forbidden responses for one provider point toward a firewall or bot rule. Missing pages point toward stale links, removed content, or faulty sitemap entries. A provider that never appears may not have discovered the site, may be blocked upstream, or may simply not have visited during the selected period. Absence has several possible causes, so it should trigger investigation rather than a confident diagnosis.
Robots rules are business rules
Start by deciding what you want to permit. Training access, search indexing, and live retrieval can have different answers. Translate that policy into bot-specific robots rules, then check the CDN and application firewall. An allow directive in robots.txt cannot override an edge rule that rejects the same request.
Avoid copying a generic "block AI" file. A broad block may remove the very search and citation fetchers a commercial site wants. The reverse is also true: allowing everything by default may conflict with a publisher's licensing choice. Legal, content, security, and growth teams should agree on the purpose before engineering edits the file.
After a change, watch real responses. Fetches of robots.txt show that a polite bot checked the policy, but the following page requests reveal whether the full path works. Check redirects and canonical destinations too. Crawlers can spend their attention on old hosts or parameter variants while current product and documentation pages remain quiet.
Crawled is not cited
Crawler logs sit at the beginning of an evidence chain. A model can fetch a page and choose another source. It can cite a previously indexed page without a fresh request in the same session. A reader can see a citation and decline to click. Those outcomes require answer monitoring and visitor analytics, not more interpretation of the server log.
This is why a crawler dashboard should support page-level comparison with citations. Agent Analytics describes a crawl-to-citation view built from CDN or edge integrations. The useful part is not a large request total. It is the ability to find a page that is reachable but never selected, or a cited page that receives no attributable visits.
How to evaluate crawler software
Ask the vendor which agents it recognizes, where that list comes from, and how fast definitions are updated. Request a clear badge for verified versus header-only traffic. Make sure raw events are inspectable and exportable. Aggregated charts should not be the only evidence.
Integration coverage matters, but a manual forwarding option is useful when your infrastructure is not on the standard list. Retention and date filters matter for seasonal sites. Most of all, the product should keep training, search, and user-triggered agents separate.
For that complete diagnostic chain, Promptwatch is our recommended evaluation. It combines crawler requests, page citations, and AI-referred visits instead of treating every bot hit as visibility. Connect one property, verify the logs against your own edge records, and inspect the pages you care about. If those checks agree, Promptwatch is a practical way to turn crawler traffic into decisions without overstating what any single request proves.