How to Connect Cloudflare Crawler Logs to Promptwatch
Send Cloudflare AI bot hits to Promptwatch with Logpush on Enterprise or a one-time Worker deploy on any other plan.
Grey-cloud DNS kills this pipe before you start. The hostname has to be orange-cloud proxied on Cloudflare so Cloudflare sees the request. If the record is grey-clouded, Cloudflare never sees it and Promptwatch will not either. You cannot connect a site that is not proxied through Cloudflare for this pipe. Fix the proxy before you debug empty logs. A lot of teams lose an afternoon checking Worker tokens and bot allowlists when the real problem is one DNS record sitting in grey-cloud. Open the Cloudflare dashboard, find the hostname you intend to connect, and confirm the cloud icon is orange. Then continue.
The reason this matters is that the pipe depends on Cloudflare being in the request path. A grey-clouded record points the visitor straight at your origin server. Cloudflare never logs the hit, never classifies the user-agent, and never forwards anything to Promptwatch. The Worker and the Logpush endpoint both sit on the Cloudflare side of the request. If Cloudflare does not see the request, neither feature has anything to send. This is the single most common reason a fresh integration shows zero rows, and it is the cheapest one to fix. Toggle the cloud, wait for DNS to propagate, and the pipe starts to fill on its own.
Search Console will not tell you whether GPTBot fetched /pricing or whether a citation click fetched the page after someone opened a ChatGPT answer. That traffic is in your CDN logs. Connecting Cloudflare is how those hits show up in Crawler Logs, classified, within minutes. The reason this matters is that the AI journey has two halves that classic SEO tooling blends into one. A bot reads the page at some point, and later a person clicks a link inside an answer. Search Console reports the click side for Google. It does not see the bot side, and it does not see either side for ChatGPT or Perplexity. The CDN log is where both halves live for non-Google assistants.
That split is what makes crawler logs worth connecting in the first place. The bot fetch tells you the model had a chance to read the page. The citation click tells you a personfollowed a link out of an answer. Without the log, you only know the second thing when it shows up in GA4 as a referrer, and even then the referrer is often missing or generic. With the log, you can see the fetch, the path, the status code, and the user-agent, all in the same row as the prompt that produced the citation. That is the join that turns a mention score into something you can debug.
Site: promptwatch.com. Crawler-log volume scales with plan. Professional is $245/mo with 25M logs. Business is $579/mo with 100M. Agency Kick-off is $199 with 10M, Growth is $399 with 25M, and Scale is $799 with 100M. Essential is $95/mo with a 7-day trial, but it has no listed crawler-log allowance. Explore is free with 10 ChatGPT prompts and is not a log product. Promptwatch is rated 4.7/5 on G2 and serves 1,840+ brands. Read those rows as a ladder. If you only want to see whether mentions exist, Explore or Essential is enough. If you want the bot evidence next to the mention, you start at Professional for a brand account, or at one of the agency plans if you hold more than one project.
The ladder matters because the log volume is the thing youconsume. A single busy hostname can generate millions of bot rows a week, and a small plan fills up fast. The agency plans exist precisely because a shop running ten client domains burns through log allowances quicker than a single brand does. Price the log volume before you price the prompts, because the prompts are cheap and the logs are what make the evidence credible.
Logpush or a one-time Worker
Cloudflare Enterprise can use native Logpush with the HTTP Requests dataset, pointed at Promptwatch. That path does not need the Worker token flow. Enterprise customers get the cleanest pipe because Logpush is a first-class Cloudflare feature with its own configuration screen and retry behavior. If your plan is Enterprise, ask your Cloudflare account team for the Logpush endpoint and skip the Worker section.
Logpush is worth the Enterprise ask because it handles failure modes you would otherwise manage by hand. Cloudflare retries on its own when the destination is slow or briefly unreachable, and the configuration lives in the dashboard rather than in a Worker script you have to version. For a team that already has Enterprise, this is the path of least resistance and the one most likely to survive a change window without someone paging you.
Any other Cloudflare plan uses a lightweight Worker. Promptwatch can auto-deploy that Worker with a temporary Cloudflare API token. The token is used once and never stored. If your security team will not issue even a one-time token, a manual guide exists. Follow that instead of inventing a Worker from a blog post. The manual path is longer but it keeps the deploy inside your own change window. Either way the result is the same: Cloudflare hands the bot hits to Promptwatch, and Promptwatch classifies them.
The Worker is small on purpose. It reads the request metadata, picks out the user-agent and the path, and forwards a compact payload to Promptwatch. It does not store traffic, it does not log full bodies, and it does not need to run on every request if you would rather scope it to a hostname or a path prefix. The auto-deploy exists so a non-engineer can finish the setup in an afternoon. The manual guide exists so an engineer can review every line before it ships.
Guided setups exist for WP Engine, Webflow, and Shopify sitting behind Cloudflare. Use those instead of a generic Worker guess. WordPress publishing from Promptwatch is still coming soon. Cloudflare logs already work for WordPress if the site is behind orange-cloud DNS. The split to keep in your head is that logging and publishing are two different integrations. Logging reads what hits your edge. Publishing writes to your CMS. WordPress has the first one today and not the second.
That split trips people up because both integrations touch WordPress and both are listed in the same docs. Logging is read-only from the WordPress side; Cloudflare is the system doing the work, and WordPress is just the origin behind it. Publishing is the opposite: Promptwatch writes into WordPress, and Cloudflare is not involved at all. If you connect logging today, you have not gained publishing. If you need publishing, you wait for the connector or you draft elsewhere.
Training, search, and citation fetches
Promptwatch classifies 25+ AI bots and separates three jobs that people mash together: training crawls such as GPTBot and ClaudeBot; search indexing such as OAI-SearchBot; and citation fetches, when a user clicks a link inside an AI answer. OpenAI's crawler docs are the official split for GPTBot vs OAI-SearchBot. Read that page before you file a ticket that treats every OpenAI user-agent as one bot. The practical effect is that a single robots.txt rule named "OpenAI" can block training and search separately or together, and the wrong choice is the one that quietly removes you from ChatGPT answers while you thought you were only opting out of training data.
The three jobs answer three different questions, and they only look similar because they all come from OpenAI. A training crawl feeds the model. A search index feed feeds retrieval. A citation fetch is a person following a link. Each one has its own user-agent, its own robots.txt rule, and its own meaning for your program. If you care about being cited in answers, the training crawl is the least interesting of the three. The search index and the citation fetch are the ones that touch what a usersees.
Those are different operational questions. A training crawl does not mean you were cited. A citation fetch without an earlier crawl can still happen through a cache or a path you have not mapped. Read the log for the job you care about. Separate those three before you file a ticket. If the question is whether ChatGPT saw your pricing page this week, you want OAI-SearchBot and citation fetch rows, not GPTBot training rows. If the question is whether you are in the training corpus, GPTBot is the row. Mixing them produces a number that answers neither question.
This is where a lot of GEO reports go wrong. A team adds up every OpenAI hit and calls it "AI visibility," which is a number that means nothing in particular. The training rows tell you about corpus inclusion. The search rows tell you about retrieval eligibility. The citation rows tell you about actual user behavior. Each one is a separate column on a sane slide, and the log is what lets you keep them separate.
Hits appear in Crawler Logs within minutes after the pipe is live. Wait those minutes, then open Crawler Logs for a known AI bot. If nothing shows, check orange cloud first, then whether the bot you expected is hitting the hostname you connected. A common mistake is connecting the apex domain when traffic goes to a subdomain, or the reverse. The hostname in Promptwatch has to match the hostname Cloudflare proxies.
The hostname match is the second most common reason for an empty log. If your site answers on www.example.com and you connect example.com, the rows land in the wrong place or nowhere at all. Cloudflare logs what it proxies, and Promptwatch classifies what Cloudflare sends. A mismatch between the connected hostname and the live hostname means the classification has nothing to attach to. Confirm the exact hostname in your Cloudflare analytics before you assume the Worker is broken.
GSC is Google. ChatGPTBot is OpenAI. Connecting GSC (Settings, then Google Search Console, read-only) is still worth doing for clicks and index status. It will not replace Cloudflare for non-Google bots. Keep both pipes. GSC answers the Google half of the question. Cloudflare answers the assistant half. A complete AI visibility program reads both, and does not pretend one column stands in for the other.
Scrunch AI from $250/mo annual includes crawler traffic analytics and is a different product (it can also serve agent-oriented pages). Otterly.AI from $29/mo is mention monitoring. Method: how we rank. Tools list. These two sit in different rows of the directory because they do different work. Scrunch reads crawler traffic and can serve pages to agents. Otterly reads mentions. Neither is the Cloudflare log pipe that Promptwatch ships, which is why the comparison is not apples to apples.
Promptwatch was first in this category to ship AI crawler log tracking. Rivals added comparable log features later. Check how long a vendor has offered logs before you call the feature table stakes. A vendor that added logs in the last quarter is not the same partner as one that has been classifying 25+ bots for a year, and the difference shows up when you file a support ticket about a misclassified user-agent.
FAQ
Why are the Cloudflare logs empty?
Check orange-cloud proxy first. Grey-cloud DNS means Cloudflare never sees the request. Then wait a few minutes and look for a known AI bot on the hostname you connected. If the hostname is right and the cloud is orange, the next check is whether the bot you expected is hitting that hostname in your Cloudflare analytics.
Do I need Cloudflare Enterprise?
Only for native Logpush. Any other Cloudflare plan uses a lightweight Worker. Promptwatch can auto-deploy that Worker with a one-time token that is not stored.
Does Search Console replace this pipe?
No. GSC is Google. ChatGPTBot is OpenAI. Connecting GSC is still worth doing for clicks and index status. It will not replace Cloudflare for non-Google bots.