How to Detect AI Agents Using or Citing My Content
Crawler logs, cited URLs and AI referral clicks are three separate signals. A buyer's checklist for tools that detect each one, and the vendor questions to ask before you sign.
"Is AI using my content?" sounds like one question. Once you start shopping for a tool to answer it, it splits into three, and most vendors only answer one of them.
An AI agent can fetch your page, an AI answer can cite it, and a person can click through from that answer. Each event leaves evidence in a different place, and none proves the other two happened.
The three signals and what each one proves
Agents fetching your pages. This lives in your server or CDN logs: a request whose user agent reads OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot or Perplexity-User. These bots do very different jobs. OpenAI's bot documentation says GPTBot collects training data, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User makes visits that a user triggered from ChatGPT or a Custom GPT. Anthropic's crawler page draws the same lines for ClaudeBot (training), Claude-SearchBot (indexing for search quality) and Claude-User (fetching when someone asks Claude). Perplexity's bot guide says PerplexityBot surfaces and links sites in its search and isn't used for training, while Perplexity-User handles fetches a user requested.
A fetch proves the page was reachable and a bot asked for it. It doesn't prove the page ended up in an answer, and it certainly doesn't prove a citation.
AI answers citing your pages. No log on your side records this. The citation happens inside ChatGPT, Perplexity or a Google AI Overview, on somebody else's screen. The only way to see it is to run the prompts yourself on a schedule and record which URLs each answer cites. A citation proves the engine used your page as a source for that prompt on that day. It says nothing about whether anyone clicked.
People clicking through. This shows up in analytics as referrals from AI assistants. For ChatGPT there's a clean marker: OpenAI's publisher FAQ says ChatGPT adds utm_source=chatgpt.com to referral URLs. A visit proves a human arrived from an AI surface. It doesn't tell you which prompt sent them, and a citation that nobody clicks leaves no trace in analytics at all.
| Signal | Where to look | What it proves | Blind spot |
|---|---|---|---|
| Agent fetch | Server or CDN logs, by user agent, checked against published IP lists | The page was reachable and a named bot requested it | Whether any answer used or cited it |
| Citation | Recorded answers to prompts you run on a schedule | An engine named your URL as a source for that prompt | Prompts you don't track; whether anyone clicked |
| Click-through | Analytics referrers, utm_source=chatgpt.com | A person came to your site from an AI answer | Citations that get no click; which prompt was asked |
Three traps in crawler data
User agents can be faked. Google says so in its crawler documentation and recommends verifying through published IP ranges or reverse DNS. OpenAI publishes an IP list per bot (openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json). Perplexity does the same for PerplexityBot and Perplexity-User (perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json) and recommends combining user-agent and IP matching in firewall rules. Anthropic warns that IP blocking may not work reliably for its bots. A tool that trusts the user agent alone counts scrapers too.
Google is the second trap. Google-Extended has no user agent string of its own. It's a robots.txt token that controls use of your content for Gemini training and grounding, and it doesn't affect inclusion in Google Search. You won't find it in a log. GoogleOther does appear in logs, but Google describes it as a generic crawler for product-team fetches, so a GoogleOther hit isn't automatically AI.
Third, the user-triggered fetchers don't necessarily follow robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic says all three of its bots respect robots.txt. So if a vendor's "robots.txt violation" report flags ChatGPT-User, that's the behavior OpenAI documents, not a rogue bot.
Vendor checklist: crawler-log tools
- Which log sources do you ingest, and does any route need a higher tier of my CDN? A Cloudflare Logpush route, for example, needs Cloudflare Enterprise.
- Do you verify bot identity against published IP lists, or match on user agent only?
- Do your reports separate training, search and user-triggered fetchers, or lump them together as "AI bots"?
- Do you show status codes and errors per bot and per page?
- Can I see which crawled pages later got cited? This one question separates log tools from visibility platforms.
Vendor checklist: citation tools
- Do you run real prompts and record the cited URLs, or estimate citations from crawler fetches? Those are different numbers.
- Do you report at page level, or only by domain? You need page level to act on anything.
- Do you capture citations of third-party pages that mention me, such as Reddit threads and YouTube videos?
- Which engines do you cover, and how often does each prompt refresh?
Vendor checklist: referral and conversion tools
- Do you report AI-referred sessions only, or conversions too?
- How is tracking installed: a script, a tag manager template, or server side?
- Can I filter AI referrals by landing page, so I can line them up with the pages that get cited?
How tools in our catalog cover the three signals
Promptwatch is the one tool here that covers all three in a single project. Agent Analytics ingests real-time crawler logs through CDN integrations (Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, or a custom HTTP endpoint), with error tracking and a crawl-to-citation path that shows whether a fetched page went on to be cited. Citation analytics comes from prompts it runs across the engines, reported at page and domain level, with Reddit and YouTube citations broken out. Visitor analytics records AI-referred visits and conversions through a lightweight script or a GTM template. Professional ($245/mo) is the first brand plan with a crawler-log allowance listed: 25M logs, alongside 150 prompts and 1M visitor events.
AthenaHQ tracks citations as part of its GEO score and lists GA4, Shopify and Webflow revenue attribution, so it speaks to signals two and three. Scrunch AI lists crawler traffic analytics next to multi-engine visibility tracking. Ask both how their data joins across signals.
Cloudflare AI Crawl Control is the free first look if your site already sits behind Cloudflare. It shows AI crawler requests by operator, status codes and most-requested paths, and lets you allow or block each crawler. On Cloudflare's Free plan it identifies crawlers by user agent only, and referral metrics need a paid plan (Pro starts at $20). It doesn't run prompts or report citations.
Known Agents is a log and referral specialist with broad sources (Cloudflare, Fastly, Akamai, AWS, Google Cloud, Vercel and others) and spoof verification. Its citation counts are estimates inferred from agent fetches, as its own llms.txt states. That makes it strong on signals one and three and indirect on signal two. It has a free plan.
Who should pick which
Pick Cloudflare AI Crawl Control if you're on Cloudflare and only want to know which bots hit you. It costs nothing extra and answers the first question.
Pick Known Agents if you want logs and AI referrals across several CDNs and have no interest in tracking prompts.
Pick Promptwatch if you need to know whether being crawled turns into being cited, and whether citations turn into visits that convert. Answering that takes all three signals joined on the same URL.
For the crawler side on its own, see our guide to GEO tools with AI crawler log tracking. To try the joined view, start at promptwatch.com.