AI crawler logs for WordPress without server access
Managed hosting often means no raw logs. Here are the ways to see which AI bots visit a WordPress site, including a plugin route and a Cloudflare route.
"We can't see server logs, we're on managed WordPress." It is one of the most common reasons teams give for not knowing whether AI crawlers visit their site. Managed hosts often hide raw access logs, or offer them in a format nobody has time to parse. Meanwhile, the questions are practical: is GPTBot reaching our product pages, is ClaudeBot hitting errors, is anything fetching the pages we want cited?
This guide covers why logs are the only reliable evidence, what your options are without server access, and where each one runs out.
Why you need logs, not guesses
Your analytics tool will not show crawler visits, because bots do not run JavaScript the way a browser does. A robots.txt file tells you what you have allowed, not what happened. And an AI answer that does not cite you tells you nothing about whether the engine's crawler could reach your page.
Only a request log shows the real picture: which bot, which URL, what response code, and when. It lets you answer three practical questions:
- Are AI crawlers reaching the pages that matter?
- Are they getting errors, redirects, or blocks?
- Did a change you made alter their behavior?
Our guide on which bots your site is open to covers the access-rules side. Logs are the evidence side.
Your options without root access
Ask your host for logs. Some managed hosts provide downloadable access logs on request. The data is real, but it is raw, delayed, and you must separate genuine bots from spoofed user agents yourself. Anyone can claim to be GPTBot in a header, so unverified counts are inflated.
Use a CDN in front of the site. If your site runs behind a CDN, its logs can show every request. Cloudflare is the common case. Logs from the edge capture requests before they reach your host, so you do not need host access.
Use a plugin. A WordPress plugin can record requests from inside the site and send them to a service. It needs no change to server or CDN configuration.
Use a measurement platform. A tool that ingests crawler data and does the bot identification for you saves the analysis step.
What changed recently
The Promptwatch changelog records two relevant changes.
On September 10, 2026, WordPress became an option in the crawler log setup. You install the Promptwatch plugin, paste an API key, and AI crawler visits start coming in without touching your server or CDN configuration.
On October 6, 2026, Cloudflare Logpush became the default crawler log setup for WP Engine, Webflow, Shopify, and Kinsta sites. That is the route for those managed platforms, including the ones that are not WordPress at all.
So the "I'm on managed hosting" answer now has two concrete paths: a plugin for WordPress, and Cloudflare for the managed platforms.
Choosing between the two
| Situation | Better route | Why |
|---|---|---|
| Self-managed or managed WordPress, no CDN control | Plugin | Install and paste a key, no infrastructure change |
| WP Engine, Kinsta, Webflow, or Shopify | Cloudflare Logpush | Edge logs, and it is the default setup for those hosts |
| Already on Cloudflare | Logpush | Reuses what you have |
| Multiple sites across platforms | Cloudflare where possible | One consistent method |
If you are not sure, start with whichever takes the least change to your stack, and switch if you find gaps. Our walkthrough on connecting Cloudflare crawler logs to Promptwatch covers the Cloudflare steps, and our overview of CDN integrations for AI crawler logs lists the other platforms.
Where Promptwatch fits
Promptwatch is our pick for this job because it handles both routes and does the interpretation. Its Agent Analytics shows real-time, IP-verified AI crawler logs, with spoofed user agents filtered out. It tracks bots by their real names, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher, and MistralAI-User.
From there you get a crawl-to-citation path with a citation rate per page, error tracking with failure explanations, stackable path filters, and CSV export. That is how you move from "bots visit" to "bots visit this page, fail on that one, and cite a third."
Integrations beyond the two above cover AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, and custom HTTP, so if you later move hosts, the data setup moves with you.
Now the honest part about plans. Crawler logs start on Professional ($245/mo). Essential ($95/mo) has no crawler logs at all. If you are on Essential, the plugin and Cloudflare routes will not help until you upgrade. The agency plans, Kick-off, Growth, and Scale, include crawler logs. So the decision includes a plan question, not only a technical one.
One more caution. Logs show what bots requested, not why an engine cited or ignored you. They are evidence for access problems, which are often the cheapest to fix, but not a guarantee of citations.
The same WordPress setup connects to publishing. Our guide to publishing GEO content to WordPress from Promptwatch explains how the official plugin handles default categories and authors.
What to look at first
Once data flows, resist the urge to read everything. Start with:
- Errors. Which AI bots receive 4xx or 5xx responses, and on which URLs? These are usually quick fixes.
- Your top commercial pages. Are they being fetched at all?
- Pages fetched but never cited. These are candidates for a content refresh.
- Spikes after changes. Did a robots.txt or plugin change alter crawl activity?
For deeper context on why some pages are visited more than others, see our post on crawl budget for AI crawlers.
Setup checklist
- Confirm your plan includes crawler logs.
- Decide between plugin and Cloudflare based on your host.
- Install, connect, and wait for the first data.
- Check that the bots you expect appear under their real names.
- Review errors first, then your key pages.
- Re-check after any hosting, CDN, or security plugin change, since those can block bots without warning.
You do not need server access to see what AI crawlers do on your WordPress site. You need one working data path and a plan that includes it.