Best GEO Software
All posts
By Best GEO Software TeamAI crawlersrobots.txtCrawler logsBuyer's guide

Which AI crawler user agents should you allow or block on your website? A buyer's guide to the policy and the tools

A buyer's guide to AI crawler user agents: which bots to allow or block, what the vendors say about robots.txt, and what log tools cost to verify your rules.

Deciding which AI crawler user agents to allow is really two purchases. One is a policy, written mostly in robots.txt and partly in your CDN. The other is a way to check that the policy does what you think, which means reading logs. Most sites get the first half roughly right and never do the second.

This guide covers both. The bot facts come from each operator's documentation, read on November 21, 2026. Tool prices come from our own directory listings.

Sort the user agents by job before you write a single rule

A list of thirty bot names is hard to reason about. Four jobs are easier.

JobExamplesWhat a block does
Search indexing for an AI answer engineOAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, ExaSearchBotRemoves or reduces your pages in that engine's answers
Classic search that also feeds AI featuresGooglebotRemoves you from Google Search, AI Overviews and AI Mode together
Fetching a page because a user askedChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Google-AgentTurns away a person's live request; robots.txt often isn't honored for these
Collecting content for model trainingGPTBot, ClaudeBot, Meta-ExternalAgent, plus the Google-Extended tokenSignals that your content shouldn't train future models

For most commercial sites the default is simple: allow the first three rows and make a deliberate, documented decision on the fourth.

What the vendors actually promise

Read the fine print, because the promises differ.

OpenAI keeps its two robots.txt controls independent. You can allow OAI-SearchBot and disallow GPTBot, and OpenAI says changes for search take about 24 hours to settle. For ChatGPT-User, it says robots.txt rules may not apply, since a user started the action.

Anthropic says its bots honor robots.txt and Crawl-delay, and that the opt-out has to be set for every subdomain. It also says blocking by IP address may not work reliably as an opt-out. Since October 30, 2026 it recognizes a machine-readable "no AI training" preference based on the IETF aipref draft, with a ClaudeBot-specific robots.txt rule taking priority over it.

Anthropic's help article listing ClaudeBot, Claude-User and Claude-SearchBot with what happens when each is disabled

Anthropic's "Does Anthropic crawl data from the web" help article, captured November 21, 2026.

Google's Google-Extended is a robots.txt token with no user agent of its own. It governs Gemini training and grounding in Gemini Apps and Vertex AI, and Google says it has no effect on Search inclusion or ranking. AI Overviews and AI Mode follow your Googlebot rules. Google also says it intends to support the AIPREF training signal (user-agent: __usage_ai-training) in early 2027.

Perplexity, like OpenAI, separates its indexer from its user fetcher, and warns that changes can take up to 24 hours. Meta says it may cache robots.txt for up to 24 hours.

What to evaluate in a crawler-log tool

Once the policy is written, you need evidence. Five questions separate the tools.

Does it verify identity? Anyone can send a request that claims to be GPTBot. OpenAI, Perplexity and Google publish IP ranges for their bots. A tool that checks against those ranges filters fakes. A tool that only reads the user-agent string counts them.

Does it read your actual stack? Logs live in a CDN, a host, or a WordPress install. Check that your source is supported before you sign.

Does it know the current bot names? Ask whether Exa's ExaSearchBot and ExaUser, Meta-WebIndexer and Google-Agent are classified. A bot list that hasn't been maintained will lump them under "other".

Does it connect a crawl to an outcome? A fetch is not a citation. The useful tools tell you whether a crawled page was then cited in AI answers.

What does it cost at your log volume? Busy sites produce a lot of bot lines.

What each option costs

Prices below come from our directory entries for each tool.

Promptwatch includes crawler logs (Agent Analytics) from Professional at $245/mo with 25M crawler logs, and Business at $579/mo raises that to 100M. Essential at $95/mo has no crawler logs, so it's the wrong plan for this job. Agency plans include crawler logs on every tier: Kick-off $199/mo (10M), Growth $399/mo (25M), Scale $799/mo (100M). Requests are checked against operators' published IP ranges, except Exa, which publishes none. Logs come in through Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint, or the WordPress plugin. Every self-serve plan has a 7-day free trial on monthly billing.

Scrunch AI Core costs $250/mo and lists AI agent and bot traffic reporting alongside 125 prompts across four engines. The crawler-side page serving it's known for (AXP) is Enterprise only, price on request.

AthenaHQ Starter is $295/mo for 3,600 credits. Its listing doesn't include a crawler-log product, so it won't answer this question on its own. Bloomreach announced on November 16, 2026 that it acquired AthenaHQ.

Cloudflare AI Crawl Control is free on every Cloudflare plan, including Free, if your site already runs behind Cloudflare. Detection beyond user-agent strings and referral metrics need a paid plan, with Pro from $20.

Known Agents has a free plan with 100K events a month, Startup at $29/mo and Growth at $299/mo. It also offers an automatic robots.txt that adds rules as new agents appear.

Promptwatch's PerplexityBot bot page describing the crawler and how its requests are matched against Perplexity's published IP ranges

Promptwatch's PerplexityBot page in its AI bots directory, captured November 21, 2026.

How to choose

If you only want to see and block bots, and you're on Cloudflare already, AI Crawl Control costs nothing and puts the block button next to the data. Known Agents is the free alternative off Cloudflare.

If the question behind the question is "are AI engines citing us, and is that worth anything," a log viewer isn't enough. You need the crawl, the citation and the visit in one place. That's the case for Promptwatch, and it's why it heads our list. Its crawl-to-citation path gives each page a citation rate, so a page that gets crawled and never cited reads as a content problem, while a page that's never crawled reads as an access problem.

A reasonable buying sequence: write the policy from the table above, then run Promptwatch's 7-day free trial on Professional with your CDN logs connected. Check that every bot you meant to allow appears, that none of them hit 403s, and that the training bots you blocked stopped arriving after the 24-hour window.