How Agencies Set Up AI Crawler Logs Across Client Sites in 2026
A practical guide for agencies connecting AI crawler logs on every client stack: why logs matter, setup paths by CDN and host, who to ask, the first-week checks, and plan volumes.
Prompt tracking tells you what AI engines say about a client. Crawler logs tell you what those engines read before they said it. For an agency, the second part is usually where the useful work hides: a pricing page GPTBot has never fetched, a moved guide that returns a 404 to ClaudeBot every day, a WAF rule nobody remembers that blocks PerplexityBot.
Reading the logs is the easy part. The real work is getting them connected on twelve client sites that run on six different stacks, each owned by a different developer. This guide covers why logs are worth the effort, the setup path for each common stack in Promptwatch, who to ask at the client, and what to check in the first week. Promptwatch details come from its llms.txt, read on 2026-10-21. Other tools' details come from their entries in our directory.
Why logs, and not just prompt data
There are three reasons, and each one changes a recommendation.
The first is the crawl-to-citation path. Promptwatch's Agent Analytics shows which crawled pages actually get cited in AI responses, and the citation rate. If a page is crawled often and never cited, the problem is the content. If it's never crawled, the problem is access or discovery. Those need different fixes, and without logs you can't tell them apart.
The second is errors. Crawl failures show up in the logs with an explanation of what went wrong. A page that returns an error to an AI crawler can't be cited from that fetch, and nobody on the client side will notice, because humans see the page fine.
The third is verification. Anyone can send a request with "GPTBot" in the user agent. Promptwatch IP-verifies crawler requests so spoofed user agents are filtered out. Without that, a scraper using a borrowed name inflates the numbers you report to the client.
It also helps to know which bot does what. OpenAI runs GPTBot, OAI-SearchBot, and ChatGPT-User. Anthropic runs ClaudeBot, Claude-User, and Claude-SearchBot. OpenAI's documentation separates training crawls, search indexing, and fetches triggered by a user's question, so a spike from one says something different than a spike from another.
Setup paths by stack
Promptwatch has a guided setup per integration. The route depends on where the client's traffic passes through.
- Cloudflare: Logpush is the default, or Promptwatch can auto-deploy a Cloudflare Worker. Traffic has to pass through Cloudflare's proxy, so a DNS-only record won't produce logs.
- WP Engine, Kinsta, Webflow, and Shopify sites behind Cloudflare: each has its own guided Cloudflare setup.
- AWS CloudFront: standard logging v2, delivered through Kinesis Data Firehose.
- Fastly: real-time HTTPS log streaming.
- Vercel and Netlify: log drains.
- Akamai: DataStream 2 edge log delivery.
- Google Cloud CDN: a Cloud Logging sink with a Pub/Sub push subscription.
- Anything else: a custom HTTP endpoint that accepts batched JSON events.
- WordPress: the official plugin covers crawler logs, visitor analytics, and publishing.
Check the client's plan tier on their own CDN or host before you promise a date. Our Known Agents listing notes that Cloudflare Logpush needs Cloudflare Enterprise and that the Vercel log drain needs Vercel Pro or Enterprise. On a lower Cloudflare tier, the Worker route is the one to plan for.
Who to ask at the client
Your contact is usually in marketing. The person who can connect logs almost never is. You need whoever controls the CDN or hosting account: an in-house developer, IT, or the outside dev shop that built the site.
Send them one short message with specific questions. Which CDN or host serves the production domain, and on what plan? Is the domain proxied through Cloudflare or only using its DNS? Can they create a log destination (a Logpush job, a log drain, a Firehose stream) or an API token for the Worker? Is there a change window or a security review for new log destinations? Ask the marketing contact to forward it with a line explaining why, because a cold request to stream production logs to a third party tends to sit in a queue.
For sites on the WordPress plugin, the marketing team can often install it themselves if they have admin access. That makes WordPress clients the fastest ones to onboard.
What to check in the first week
- Confirm data is arriving. Give it a day or two for the first AI crawler requests to show up. If nothing appears, recheck the proxy setting or the log destination before you look anywhere else.
- Compare verified bot traffic with what the client's old analytics claimed. If an earlier report counted every "GPTBot" user agent, expect the verified number to differ.
- Read the errors. Look for 404s on moved pages, server errors, and requests blocked by security rules. These are often the quickest wins you'll find all quarter.
- Line up top crawled pages against the pages the client wants cited. A gap here is the most common first finding.
- Open the crawl-to-citation path for the priority pages and note the citation rate as a baseline.
- Check robots.txt against the bots the client actually wants to allow. Promptwatch's free directory of 440+ AI crawlers lists each bot's purpose, robots.txt behavior, and allow or block guidance.
Then write a one-page baseline for the client. Next month's report needs something to compare against.
Volume per plan
Crawler logs are a plan allowance. On the agency plans, Kick-off ($199 a month) includes 10M crawler logs, Growth ($399) includes 25M, and Scale ($799) includes 100M. Agency plans have unlimited projects but a single crawler-log figure on the plan card, so plan as if every connected client site draws from that one number. On brand plans, Professional ($245) includes 25M and Business ($579) 100M. Essential has no listed crawler-log allowance.
To size it, ask each client's developer for a rough monthly count of AI bot requests. Sites already on Cloudflare can read that from AI Crawl Control, which shows request volume by crawler on every Cloudflare plan at no cost.
How other tools compare
A handful of tools in our directory document crawler or bot analytics. Most prompt trackers don't, and some listings say so outright: AthenaHQ's entry notes no crawler analytics, and SEOmonitor and Semrush AI Toolkit have no published crawler-log product.
| Tool | Log sources documented | Bot verification | Crawls tied to citations |
|---|---|---|---|
| Promptwatch | Cloudflare, CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, custom HTTP, WordPress | IP-verified | Crawl-to-citation path |
| Known Agents | WordPress, Cloudflare, Fastly, Akamai, CloudFront, Google Cloud, Vercel, Node.js, REST API | Spoof checks; IP or message signature where supported | Estimated from fetches |
| Cloudflare AI Crawl Control | Cloudflare traffic only | User agent on Free; deeper on paid plans | No |
| Honeyb | Forwarded server logs; CDNs not named | Reverse DNS and IP | Joined to citations |
| Proofsource | Cloudflare or a log drain | IP ranges or reverse DNS | Crawled versus cited pages |
| Surva.ai | Nginx or Apache uploads, Cloudflare Worker | Not documented | Not documented |
| Scrunch AI | Not documented | Not documented | Not documented |
| DemandSphere | Enterprise add-on (Analytics AX) | Not documented | Not documented |
Known Agents has the broadest source list after Promptwatch and a free plan, but it doesn't run prompts, so it can't tell you what AI says about the client. Cloudflare AI Crawl Control is free and useful for sizing, but it only sees Cloudflare traffic. Honeyb and Proofsource both verify bots and connect crawls to citations, though their listings name few or no specific CDN integrations. Surva.ai puts crawler logs on Growth at $119 a month. Scrunch AI lists crawler traffic analytics without log-level detail on its listing, and DemandSphere sells log analytics as an Enterprise add-on with per-GB storage fees.
The recommendation
The deciding factor for an agency is coverage across stacks. A log tool that only reads Cloudflare leaves out the client on CloudFront and the one on Vercel. Promptwatch's Agent Analytics has a documented path for each of those, plus a custom HTTP endpoint for the odd ones, and it ties every verified crawl to the citations that follow in the same project as the client's prompts. Start on Kick-off or Growth at Promptwatch, connect your easiest WordPress or Cloudflare client first, and see the Promptwatch review for the plan details.