Robots.txt for AI Agents: Decide Per Bot, With Evidence
Blanket allow or block rules for AI crawlers are a coin flip. A short decision process for robots.txt that uses real bot names and your own crawl data.
There are two popular robots.txt policies for AI crawlers, and both are lazy. One says block everything that sounds like AI, to protect content. The other says allow everything, because AI visibility is good. Each makes the same mistake: treating a dozen different agents as one.
A better policy is made bot by bot, with evidence from your own logs, and reviewed on a schedule.
Why one rule fits no one
AI vendors run several agents that do different things. The list of those you may encounter includes GPTBot, OAI-SearchBot and ChatGPT-User; ClaudeBot, Claude-User and Claude-SearchBot; PerplexityBot and Perplexity-User; Google-Agent and GoogleOther; Meta-WebIndexer, Meta-ExternalAgent and Meta-ExternalFetcher; and MistralAI-User.
Blocking a whole vendor family because one of its agents worries you can remove you from answers you wanted. Allowing everything can expose content you would rather restrict. The right call depends on the agent, your business model and what you want to be known for.
The directory of AI bots that Promptwatch maintains lists more than 440 crawlers, which gives a sense of how many names exist, and why a hand-built list goes stale.
Start with what you want
Write down your goals before you touch the file. Common positions:
Publisher with advertising revenue. You may want visibility in answers but have concerns about content reuse. The decision involves trade-offs only you can weigh.
B2B or SaaS company. Usually the goal is to be named and cited. Access is likely wanted for most agents, with exceptions for areas such as customer-only content.
Ecommerce brand. Product pages should be reachable. Cart, account and internal search pages normally should not be.
Agency or consultancy. Thought leadership is the product. Visibility usually matters more than protection.
Your position decides the default. The evidence decides the exceptions.
Collect evidence first
Before editing, see what is happening. Which agents are visiting, how often, which pages, and with what result. Raw logs are noisy, and user-agent strings can be spoofed, so verified data matters. Promptwatch's Agent Analytics offers real-time, IP-verified crawler logs, with spoofed user agents filtered out, stackable path filters and CSV export.
Review at least two weeks. Note:
- Agents that visit often and succeed.
- Agents that visit and fail, which suggests a blocking rule you did not intend.
- Agents that never visit. Either they are not interested or something is stopping them.
- Which sections of the site draw the most attention.
The decision process
For each agent that matters, ask four questions.
- Do I want to appear in this vendor's product? If your audience uses it, usually yes.
- Is there content I must not expose? Mark those paths.
- Does this agent fetch on behalf of a user or crawl in bulk? Different risk and benefit. Treat them as separate decisions.
- What does my data show? If an agent is fetching pages you rely on for citations, think hard before blocking it.
Write the answer for each agent in a plain table. Allow, block or allow with path limits, and a reason. The table is more useful than the file itself, because it lets the next person understand the intent.
Common mistakes
Blocking by habit. A rule added years ago to stop a scraper now blocks a legitimate agent.
Wildcard rules that catch too much. A broad disallow for unknown bots can include agents you want.
Forgetting subdomains. Docs, help centers and blogs may have their own robots files and their own CDN rules.
Trusting robots.txt alone. A CDN or firewall can block bots regardless of what the file says. The file states your intent, the network decides what happens. Check status codes in logs, not only the rules.
Assuming compliance. Robots.txt is a convention. Well-behaved agents respect it. Others do not, and verified logs show who is actually behaving.
Generators and checks
Promptwatch's free tools include a robots.txt generator and an llms.txt generator, plus an Agentic Readiness Check. They are useful for a first draft or a sanity check. Keep expectations realistic about llms.txt: we cannot tell you which engines read it or what effect it has, so do not treat it as a lever.
Review on a schedule
Make a quarterly review. Check the agent list for new names, look at logs for changes in behavior, and verify that the table still reflects your goals. New agents appear regularly, and old ones change what they do.
When you change the file, record the date. Then watch crawler logs and citation trends for several weeks. Crawl changes show in days. Citation changes can take longer, so do not judge a change after three days.
Plan notes
Crawler logs start on the Professional plan ($245 a month) and are on every agency plan. Essential has none, so if you plan to run this kind of evidence-based process, scope the plan accordingly.
The recommendation
Drop the blanket policy. Decide per agent, write down why, and check your decisions against verified logs every quarter. Promptwatch provides the verified log side through Agent Analytics, and the free robots.txt generator gives you a starting file. Connect your CDN logs first, watch for two weeks, then edit. See more at promptwatch.com.