How do I know if AI crawlers are visiting my site? A buyer's guide to crawler tracking tools
Buyer's guide to seeing which AI crawlers visit your website and what content they read: the options from free CDN dashboards to full platforms, costs and how to choose.
You can answer "are AI crawlers visiting my website?" for free, in an afternoon, with a log export and a spreadsheet. Answering it every week, for every page, with bots verified and tied to what happens next, is where software earns its fee. This guide is for teams deciding how far along that line to go.
Three ways to find out
Read the raw logs yourself
Export your server or CDN access logs and search for the bot names the AI companies publish: GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI; ClaudeBot, Claude-SearchBot and Claude-User from Anthropic; PerplexityBot and Perplexity-User; Meta-WebIndexer and its siblings; Google-Agent. It costs nothing but time. The weak spots are verification (anyone can type "GPTBot" into a user agent) and repetition, since you'll be doing it again next month.
Use the dashboard your CDN already has
If you're on Cloudflare, Cloudflare AI Crawl Control is included on every plan, Free included. It groups AI crawler requests by operator and shows status codes, data transfer, most-requested paths, content types and robots.txt violations, and you can allow or block a crawler from the same screen. It only sees traffic that passes through Cloudflare, identifies bots by user agent alone on the Free plan, and reports no prompts or citations.
Buy a platform that joins logs to AI visibility
The third option ingests your logs, verifies each bot, and puts crawler activity next to prompt tracking, citations and AI-referred traffic. That join is the reason to pay. A crawl on its own is a curiosity. A crawl followed by a citation, or a crawl that keeps failing on the page you most need cited, is something you can act on.
What to evaluate
Ask these five questions in every demo.
Which log sources does it accept? Your stack decides this. Promptwatch lists Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel and Netlify log drains, Akamai DataStream 2, Google Cloud CDN, a custom HTTP endpoint and a WordPress plugin, with guided setups for WP Engine, Kinsta, Webflow, Shopify and HubSpot behind Cloudflare.
How does it prove a bot is real? Look for checks against the IP ranges operators publish. OpenAI and Perplexity both publish theirs, and Google warns on its own crawler page that user agents can be spoofed.
Does it separate training, search and user fetches? These mean different things for you, and a single "AI bots" total hides that.
Can you see the pages and the errors? Top paths per bot, status codes, and filters by path are what answer the "what content are they reading" half of the question.
Does it connect crawls to outcomes? Citations in AI answers, and visits and conversions on your own site.

Promptwatch "Analytics for AI agents" feature page with sample dashboard panels, captured November 8, 2026.
What the options cost
All figures come from our directory entries or Promptwatch's published plans.
- Raw logs: free, plus analyst time.
- Cloudflare AI Crawl Control: free on any Cloudflare plan. Referral metrics and detection beyond user agent strings need a paid plan, with Pro from $20 a month.
- Known Agents: a free plan with 100K events a month and one project, then paid tiers up to Growth at $299 a month and custom Enterprise. It covers agent analytics with spoof verification, AI chat referral tracking and an automatic robots.txt, but no prompt tracking.
- Promptwatch: crawler logs start on Professional at $245 a month (25 million crawler logs) and Business at $579 (100 million). Agency plans include them from Kick-off at $199 (10 million). The Essential plan has no crawler logs. There is a 7-day free trial on monthly billing.
- Scrunch AI: Core is $250 a month. The product includes AI agent and bot traffic reporting; our entry doesn't say which plan carries it, so ask.
- Ahrefs Brand Radar: reports on AI crawler activity next to its AI visibility data. Custom Prompts come with paid Ahrefs plans and the AI Visibility Index starts at $199 a month.

Ahrefs "Bot Analytics" page, captured November 8, 2026.
How to choose
Match the tool to the question you actually need answered.
If you only want to know whether AI bots reach you at all, and you're on Cloudflare, use AI Crawl Control and stop there. If you're not on Cloudflare, a one-off log export does the same job.
If you need to control bot access across a messy stack, with robots.txt kept current as new agents appear, Known Agents fits that brief. Budget for its event model: every API call counts as an event, and its own guidance is to plan for roughly twice your human visits, so a busy site will outgrow the free tier quickly.
If the real question is "are AI crawlers reading the pages I need cited, and is it working?", you need logs and AI visibility in one product. That is Promptwatch on Professional or above: IP-verified logs, a crawl-to-citation path with a citation rate for each page, and visitor analytics that show the AI-referred traffic that follows. Connect your CDN during the free trial and you'll have a week of verified crawler data before the first invoice.