AI Crawler Visibility Tool for Bot Activity on Your Website: 'AI Crawlers' in Cloudflare, Ahrefs, Semrush (2026)
Cloudflare, Ahrefs and Semrush all have an AI crawlers screen now. They measure different things. Here is what each one sees, and the join none of them make.
Search for an AI crawler visibility tool and three familiar names come back: Cloudflare, Ahrefs and Semrush. Each now has a screen labeled something like "AI crawlers" or "AI bots." They aren't measuring the same thing. One reads traffic at the network edge, one reads your server logs, and one mostly simulates a crawl and parses your robots.txt. Picking between them is easier once you know which kind of data you're looking at.
This buyer guide asks four questions of every tool, then shows what all three leave out.
Four questions to ask any crawler tool
Where does the data come from? Edge traffic, server or CDN logs, a simulated crawl, or a robots.txt check. Only the first two tell you what real bots actually did.
Which bots does it name? "AI bots" as one bucket is less useful than GPTBot, OAI-SearchBot and ChatGPT-User shown separately, since OpenAI documents them as different robots doing different jobs (training, search and user-initiated fetches).
What does it cost to get the useful view? Several of these features sit on a specific plan or beta.
Does the crawl connect to anything? A prompt, a citation in an AI answer, a visit that converted. This is the question that separates crawler monitoring from AI visibility.
Cloudflare AI Crawl Control
AI Crawl Control, formerly AI Audit, is available on all Cloudflare plans. Its Overview, Crawlers and Metrics tabs show requests by crawler and by operator (OpenAI, Microsoft, Google, ByteDance, Anthropic, Meta), allowed versus unsuccessful requests, robots.txt violations, status codes, paths and hosts. It is also the only tool here that acts on the traffic: you can allow or block each crawler, and pay per crawl is in private beta.
The free plan detects bots by user-agent string only and limits the Metrics tab to the past 24 hours. Referral counts need a paid plan. Enterprise with Bot Management adds detection IDs and configurable timeframes, and a GraphQL Analytics API exposes the data. What it doesn't do is connect a crawl to a prompt, a citation, or a converting visit.
Ahrefs Bot Analytics
Bot Analytics is server-side bot analytics from Ahrefs, free while it's in beta. It sorts traffic into 12 bot categories, including AI assistants, search engines and SEO tools, and gives a per-bot view for crawlers like Googlebot, GPTBot and ClaudeBot. You see pages, status codes and site structure, with an "AI bots" toggle to isolate the AI side.
Logs come in through Cloudflare Logpush on Enterprise, a Cloudflare Worker on any plan including free, a Vercel log drain, Fastly, or an Amazon data stream option. Note that Bot Analytics is a separate product from Ahrefs Web Analytics and from Ahrefs Brand Radar, the AI mention index. Owning one doesn't mean the crawl data shows up in the other.
Semrush: three different crawler features
Semrush has more than one answer here, and they're easy to confuse.
- Site Audit gives an AI Search Health score and a "Blocked from AI Search" widget showing which of eight major AI crawlers your robots.txt blocks. There's an "AI Search" issues filter, and you can set the audit's user agent to OpenAI-Search. This is a simulated crawl plus a rules check. It's useful for catching a bad robots.txt line, but it isn't a record of real bot visits.
- Log File Analyzer reads real data. You upload or connect server logs and filter for GPTBot, ChatGPT-User, OAI-SearchBot and ClaudeBot.
- Semrush Enterprise adds Bot Analytics in Site Intelligence (30 bots), an "Agent Analytics" view inside its AI Optimization product, and Crawler Profiles that simulate ChatGPT (ChatGPT-User, OAI-SearchBot, GPTBot) and Googlebot crawls. It's Enterprise-only with no public price.
A naming warning: Promptwatch's crawler product is also called Agent Analytics, and per Promptwatch's founders it shipped first in the category. Different products, same name. Make sure a sales call is talking about the one you mean. Semrush's AI answer tracking is a separate add-on, covered on our Semrush AI Toolkit page.
Side by side
| Tool | Data source | Bots named | Access | Connects crawl to answers or visits? |
|---|---|---|---|---|
| Cloudflare AI Crawl Control | Edge traffic | By crawler and operator | All plans; free plan 24h metrics, user-agent detection | No |
| Ahrefs Bot Analytics | Server and CDN logs | 12 categories, per-bot views | Free during beta | No |
| Semrush Site Audit | Simulated crawl, robots.txt | 8 AI crawlers (block check) | Semrush; plan not covered here | No |
| Semrush Log File Analyzer | Uploaded or connected logs | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot | Semrush; plan not covered here | No |
| Semrush Enterprise | Logs and simulated crawls | 30 bots | Enterprise, no public price | Not published |
| Promptwatch Agent Analytics | CDN and server logs | ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther, Meta AI crawler | Professional $245/mo and up; agency plans from $199/mo | Yes: prompts, citations, visitor conversions |
The join none of the three make
Every tool above can tell you a bot arrived. None of them, on what they publish, can tell you whether that visit led to your page being cited in an answer, or whether the answer sent someone who bought.
That's the gap Promptwatch fills. Agent Analytics ingests real-time crawler logs from Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN or a custom HTTP source. It shows the crawl-to-citation path and tracks bot errors. Because the same account runs prompt tracking across ChatGPT, Gemini, Claude, Perplexity, Copilot, Google AI Overviews and more, plus citation analytics down to the page, a crawl of a page sits right beside the prompts where that page got cited. Visitor analytics, installed with a script or a GTM template, then adds the AI-referred visits and conversions. So if a citation drops, you can check whether the bot stopped fetching the page, started getting errors, or kept crawling while the model picked a competitor.
On Cloudflare, Enterprise accounts connect through Logpush. Other plans get an auto-deployed Worker, and the record must be orange-cloud proxied. Crawler logs begin on Professional ($245/mo, 25M logs) and Business ($579/mo, 100M). Essential at $95/mo has visitor analytics but no listed crawler-log allowance. The agency plans include them from Kick-off ($199/mo, 10M).
Who should pick which
Keep Cloudflare AI Crawl Control on regardless. It's free and it's where you block. If you want a free look at real bot traffic and already use Ahrefs, Bot Analytics during its beta is a good second opinion. If your worry is a robots.txt mistake, Semrush Site Audit catches that quickly. If you're on Semrush Enterprise, its bot views are worth a demo.
If the question is whether AI crawling is turning into citations and revenue, though, you need the crawl joined to the prompt and the visit. Promptwatch is the one we'd evaluate first for that. Our Cloudflare setup guide shows the connection in a few steps.