Best GEO Software
All posts
By Best GEO Software Teamcrawler logsclaudemetaagent analyticsdata reportbuyer's guide

Crawler-log features checklist: Claude's citation crawler and Meta-WebIndexer

Two Promptwatch crawler reports show how fast AI bot traffic shifts and how much counting methods matter. A feature checklist for buying AI crawler-log software.

Two of Promptwatch's public crawler reports describe very different events, and both are useful for a buyer for the same reason: they show how quickly the mix of AI bots hitting a website can change, and how much the numbers depend on how bots are identified and counted. If you are choosing software to read your AI crawler logs, those two things are what you're paying for.

This post goes through each report, what its method can and can't tell you, and then the checklist we'd use to judge any crawler-log product.

Claude's citation crawler: from about 30 visits a day to thousands

Promptwatch's Claude citation crawler report tracks Claude's daily share of all tracked AI citation crawler traffic from December 11, 2025 to April 19, 2026, shown as a seven-day rolling average. In mid-December the crawler made "roughly 30 visits a day (about 0.04%)". By mid-April it was "several thousand a day", which the page calls "more than 100x in four months". In late March, daily visits "jumped from the hundreds into the thousands within two weeks."

The stat cards give the scale. Claude peaked at 1.73% of tracked citation crawler traffic on April 13, 2026, was at 1.09% on April 19, and averaged 0.45% over the 130 days, per the same report. The page keeps it in proportion: Claude "is still around 1% of tracked citation crawler traffic, with ChatGPT dominating the rest."

Promptwatch line chart of Claude's citation crawler share of AI crawler traffic from December 2025 to April 2026, flat at first and climbing sharply from late March Promptwatch's "Claude Citation Crawler Share of AI Crawler Traffic, April 2026" chart (Dec 11, 2025 to Apr 19, 2026, seven-day rolling average), captured November 14, 2026.

Two method notes matter for buyers. The report doesn't name which user agent it counts as Claude's citation crawler, and it doesn't mention IP verification. And it predates Promptwatch's own November 4 reclassification of crawlers, after which search indexers such as Claude-SearchBot count as AI crawls and only bots that fetch a page while answering a user count as citation crawls. So "citation crawler" in an April report may not mean exactly what it means in the product today.

Meta-WebIndexer: from 1 in 50 to more than 1 in 3

The Meta-WebIndexer report is a sharper event. Meta-WebIndexer made up about 2.2% of all AI crawler requests in mid-July 2026 and hovered around 2% until July 17. It spiked to about 23% between July 20 and 22, then surged from August 5, was "climbing past 26%" on August 6, and peaked at 37.8% on August 9, "roughly 17x increase in under a month." The page describes that as going from "about 1 in 50" to "more than 1 in 3" of AI crawler requests (daily series, July 11 to August 9, 2026, with August 10 a partial day).

Promptwatch line chart of Meta-WebIndexer's daily share of AI crawler requests from July 11 to August 9, 2026, with a first spike in late July and a steep second ramp in August Promptwatch's Meta-WebIndexer daily share chart (Jul 11 to Aug 9, 2026), captured November 14, 2026.

Here the method is spelled out, and it's different from what you'd want in a product. The share counts only the meta-webindexer user agent, divided by requests from all AI crawlers Promptwatch tracks, "identified by user agent". The page doesn't say the requests were IP-verified. Googlebot, Bingbot, link-preview fetchers like FacebookExternalHit, and human visitors are excluded entirely, and Meta's other crawlers count in the denominator but not the numerator. The motive the page discusses, that Meta is building its own search index, rests on reports and a tweet that says "allegedly", so treat it as a possibility rather than a fact.

We wrote about each event on its own in Claude's citation crawler and Meta-WebIndexer in AI crawler logs. Here they serve as test cases.

What the two reports teach a buyer

Our interpretation, drawn from the two methods side by side:

First, user-agent strings can be faked. A report that counts by user agent, like the Meta one, is fine for spotting a trend across many sites. For your own site, where you may make blocking decisions or explain a hosting bill, you want requests checked against the operator's published IP ranges.

Second, bot lists go stale. Claude's citation crawler barely registered in December 2025 and was making thousands of visits a day by April 2026 (Claude report), and Meta-WebIndexer went from about 2% to over a third of AI crawler requests between mid-July and August 9, 2026 (Meta report). If your tool's bot list doesn't include a new crawler, you won't see the surge at all.

Third, the difference between an indexing crawl and a citation crawl changes what you should do. A surge in an index crawler means your pages may be going into a new search index. A rise in a citation crawler means pages are being fetched while a user waits for an answer. They call for different checks.

The crawler-log features checklist

  1. IP verification. Requests matched against each operator's published IP ranges, with spoofed user agents filtered out, and a clear label on bots that can't be verified.
  2. Current bot coverage. At minimum GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, Meta-WebIndexer, Meta-ExternalAgent, and Meta-ExternalFetcher, with new bots added as they appear.
  3. Index crawls separated from citation crawls.
  4. A crawl-to-citation path: for each page, how often it is crawled and how often it is then cited. That tells an access problem from a content problem.
  5. Error tracking with explanations, so you can see when a bot hit a 403 or a timeout and why.
  6. Host checks, flagging logs from hostnames that aren't your site.
  7. Integrations for where your logs actually live: CDN, host, or CMS.
  8. Volume that fits your traffic, stated per plan.
  9. Export, so your engineering team can check the numbers.

Questions for the vendor

  • "Do you verify crawler requests by IP? Which bots can't you verify, and how do you label them?"
  • "When did you last add a new AI crawler to your list?"
  • "Do you separate crawlers that build an index from crawlers that fetch pages during an answer?"
  • "Can you show me, for one page, its crawls next to its citations?"
  • "How do I connect logs if my site is on Cloudflare, CloudFront, Vercel, or WordPress?"

How Promptwatch's Agent Analytics answers the checklist

Promptwatch Agent Analytics covers each item on that list. Crawler logs are real-time and IP-verified against each operator's published ranges, so spoofed user agents are filtered out. The tracked bots include all of those in item two, plus MistralAI-User and Exa's ExaSearchBot and ExaUser since November 6; the Exa bots are labelled as not verified, because Exa publishes no IP ranges. Since the November 4 reclassification, index crawlers such as OAI-SearchBot, Claude-SearchBot, and Meta-WebIndexer count as AI crawls, and only answer-time fetchers such as ChatGPT-User count as citation crawls.

The crawl-to-citation path gives a citation rate per page. Error tracking explains failures, filters stack and persist, and logs export to CSV. Since November 13, a Host column flags logs from hostnames that aren't the project website. Logs come in through Cloudflare (Logpush or an auto-deployed Worker, with guided setups for WP Engine, Kinsta, Webflow, Shopify, and HubSpot), AWS CloudFront, Fastly, Vercel and Netlify log drains, Akamai DataStream 2, Google Cloud CDN, a custom HTTP endpoint, or the WordPress plugin.

Promptwatch Agent Analytics page describing its AI crawler log tracking and crawl-to-citation analysis Promptwatch's Agent Analytics page, captured November 8, 2026.

Crawler logs start on Professional at $245 a month, with 25M logs; Business ($579) includes 100M. Essential ($95) has no crawler logs. Agency plans include them from Kick-off ($199, 10M logs). For a quick robots.txt check before you buy anything, Promptwatch's free Agentic Readiness Check tests robots.txt against nine AI crawlers.

How others compare in our catalog: Scrunch AI lists AI agent and bot traffic reporting on Core ($250 a month) alongside its page-serving layer for crawlers; ask whether it verifies by IP. Ahrefs Brand Radar reports on AI crawler activity inside Ahrefs. Our AthenaHQ entry lists no crawler analytics. None of these entries describes IP verification or a crawl-to-citation rate per page.

Bottom line

Claude's citation crawler grew from about 0.04% to a 1.73% peak of tracked citation crawler traffic between December 2025 and April 2026 (Claude report), and Meta-WebIndexer went from about 2% to 37.8% of AI crawler requests between mid-July and August 9, 2026, counted by user agent (Meta report). Shifts like these are why IP verification and a current bot list are the first things to check. Promptwatch's 7-day free trial on monthly billing covers Professional, which is enough to connect your logs and see which AI crawlers really visit you.