AI Crawler Logs in Agent Analytics: Crawl-to-Citation Path and Error Tracking

Google Search Console will tell you how Overviews and AI Mode treated your pages. It will not tell you whether ChatGPTBot fetched /pricing last night, whether ClaudeBot got a 403, or whether a crawl ever turned into a citation. That pipe is crawler logs. Agent Analytics is the function that reads those logs and joins them to answers, and the join is the part a spreadsheet cannot reproduce. The gap is the point. GSC covers Google's surfaces. The crawlers that feed ChatGPT and Claude are other companies, and their fetches live in your CDN logs, not in GSC. Without the join, a miss on ChatGPT is a mystery. With the join, a miss on ChatGPT is either a blocked fetch or a content problem, and you know which.

Promptwatch is the tool that has it. Agent Analytics logs ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther, Google-Agent, and Meta's AI crawler. You get crawl discovery, crawl-to-citation rate, error tracking, CSV export, path filters, and sitemap intelligence. Logs come in through Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, or custom HTTP. Site: promptwatch.com. The CDN list matters because the logs already exist where you serve traffic. You do not install a new tag. You connect the CDN you already use, and the crawls you are already receiving become the data.

What the log shows that GSC does not

GSC generative reports cover Google's own surfaces. ChatGPT Search and Claude are other companies, and their fetches live in their own logs, not in GSC. A blocked GPTBot rule can kill citations while GSC still looks fine, because GSC never saw the GPTBot fetch in the first place. Keep GSC. Do not file a ChatGPT miss under it, because the report will not contain the answer you need. This is the most common misdiagnosis in GEO. A team sees a citation drop, opens GSC, finds nothing wrong, and concludes the content is the problem. The content is not the problem. The fetch is the problem, and the fetch is in a log GSC does not carry.

Crawl discovery is pages found, indexed, or blocked by those bots. Top pages is which paths they hit most, which tells you where the bots are spending their attention. Path filters, exact or partial, let you stare at /blog without the marketing site, and the filters persist as you move between log views so you do not rebuild the filter every screen. The persistence is a small thing that compounds. When you are moving between crawl discovery, top pages, and error tracking, a filter that survives the move is the difference between reading the log and fighting the log.

Crawl-to-citation and errors

The useful join is crawl-to-citation: which crawled pages appear as sources, and what your citation rate is on that join. A page that is crawled weekly and never cited is a fetch without a payoff, and it is a ticket for the content team rather than the engineering team. A page that is cited but never in the log may be reached through a cache, a partner, or a bot you have not connected yet, and that is a different investigation. The two cases look similar from the outside, a page that is and is not cited, and they need different teams. The join is what tells them apart. Without it, you send the content team to rewrite a page that was never fetched, or you send engineering to fix a robots rule on a page that was fetched fine and just was not good enough to cite.

Error tracking lists the failures that stop a fetch: status codes, blocks, broken paths, with a readable reason. Export CSV when you want the log in a warehouse, or when a ticket needs the raw rows attached. Sitemap intelligence supports multiple sitemaps per project and surfaces page-level prompt opportunities from those URLs, so discovery is not only whatever the bot guessed at. The readable reason is the part that saves time. A 403 in a raw log is a number. A 403 with "blocked by robots rule on /staging" is a ticket you can assign. The export is for the case where you need to join the log to your own warehouse data, or where a ticket needs evidence attached.

Promptwatch was first in this category to ship AI crawler log tracking. Most rivals added something comparable about a year later. Check how long a competitor has offered logs before you treat the feature as table stakes, because a feature that shipped last quarter is not the same as one that has been joined to citations for a year. The join is the work. A log without the join is a list of fetches. A log with the join is a diagnosis. The year of head start is a year of the join being wired to citation analytics, which is not the same as a year of storing rows.

Volume and who else has a piece

Crawler-log allowance scales with plan. Professional is $245/mo with 25M logs. Business is $579/mo with 100M. Agency Kick-off is $199/mo with 10M, Growth is $399/mo with 25M, and Scale is $799/mo with 100M. Essential is $95/mo and has no listed crawler-log allowance. Explore is free with 10 ChatGPT prompts and is not a log product. The allowance matters because logs are high volume. A busy site receives millions of crawler hits a week, and a small allowance fills up before the month ends. The 25M tier on Professional is the realistic floor for a site that publishes regularly. The 100M tier on Business and Scale is for high-traffic or multi-client work.

Scrunch AI from $250/mo annual includes crawler traffic analytics and refreshes weekly. Otterly.AI from $29/mo is monitoring only. Profound Starter is $99/mo annual and ChatGPT-only. Method: how we rank. Tools list. Scrunch has the crawler traffic analytics, which is the closest comparable surface, but it refreshes weekly and it assumes engineering to serve crawler-specific HTML. Otterly is monitoring only, which means it tells you whether you appeared, not whether you were fetched. Profound Starter is ChatGPT-only, which means it does not carry the multi-crawler path on the entry tier.

Who should pick which

Pick Promptwatch Professional if you want crawl-to-citation on a brand plan and 25M logs is enough volume. Pick Business if you need 100M logs. Pick an agency plan if you run multiple projects and want unlimited prompts with a log allowance. Pick Scrunch if you want crawler traffic analytics on a weekly cadence and have engineering to serve crawler-specific HTML. Pick Otterly if you only need a mention monitor and a cheap start. Pick Profound Starter only if ChatGPT-only coverage fits, because it has no crawler path on our listing. The picks split by surface and cadence. A daily join needs Promptwatch. A weekly analytics read can use Scrunch. A cheap mention check can use Otterly. A ChatGPT-only entry can use Profound Starter.

FAQ

Can I paste Cloudflare logs into a spreadsheet instead?

Yes. You will not get crawl-to-citation on the same prompt next to the 403. That join is the product. The spreadsheet gives you the rows. It does not give you the diagnosis.

Is Google-Agent the same as Googlebot?

No. Google-Agent and GoogleOther are the AI-oriented variants Promptwatch logs. Classic Googlebot still belongs in GSC and your regular crawl reports.

What to do this week

Confirm GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot are allowed on the pages you want cited, because a blocked bot is the most common silent miss.
On Professional, Business, or a self-serve agency plan, connect the CDN you already use to Promptwatch Agent Analytics.
Filter logs to one money path and list errors from the last seven days, so the first read is small enough to act on.
Compare crawled URLs to cited URLs. Ticket pages that are fetched and never sourced to the content team, not engineering.
Upload or sync the sitemap so discovery is not only whatever the bot guessed.