Technical Optimization for AI Crawlers: Schema, Crawlability, and Robots.txt

You cannot win a citation on a page the bot cannot fetch. Technical optimization for AI crawlers is crawlability, robots.txt, and structured data that matches the HTML. It is not a secret Overview schema type, and it is not an llms.txt file that Google promised to honor. Google never made that promise, and the docs that get cited for the file are not Google's. The order of operations is the point. Fetch first, because a page that is not fetched cannot be cited. Then content, because a fetched page still needs a claim worth citing. Then schema, because schema describes the content to the model. The file that does not exist and the schema type that does not exist are the two most common GEO detours, and neither produces a citation.

Promptwatch is the tool that runs crawlability analysis, suggests schema fixes, and includes a free llms.txt generator plus a robots.txt generator aimed at AI bots. Essential at $95/mo with a 7-day trial already includes technical optimization. Explore is free with ChatGPT and 10 prompts. Site: promptwatch.com. The free generators are tools, not recommendations. A generator that writes a file is not a claim that the file changes rankings, and the robots.txt generator is the one that needs a human read before deploy.

Crawlability and robots.txt

Crawlability analysis asks whether ChatGPTBot, ClaudeBot, PerplexityBot, and the Google AI variants can reach the page you care about. Agent Analytics is the log side of the same question, and it tells you what happened rather than what should happen. It starts on Professional for brand plans and is also included on agency plans. Technical optimization is the "what to change" side: blocked paths, errors, thin or unreadable responses. The two sides answer different questions, and a healthy crawlability report can still hide a 403 in the logs. The crawlability report is the theory. The log is the practice. A page that should be reachable can still return a 403 to a specific bot because of a WAF rule, and only the log shows that. A team that reads only the crawlability report will conclude the page is fine while the bot is being blocked at the edge.

Robots.txt is still the allow and deny list. The user agents that matter for AI search are GPTBot, OAI-SearchBot for ChatGPT Search, PerplexityBot, and ClaudeBot. Blocking GPTBot "to save crawl budget" while asking why ChatGPT never cites you is a self-own, because the block is the cause. Promptwatch's robots.txt generator for AI writes a starting file. Read it before you deploy. A generator will not know your staging directory, and a blanket allow copied into production can expose paths you did not intend to publish. The crawl-budget argument is the old SEO reflex applied to a new surface, and it inverts the goal. The goal is to be cited, which requires being fetched, which requires being allowed. Blocking the bot to save budget saves the bot from ever reading you.

Schema without the Overview myth

Google's AI features documentation and AI optimization guide are the source on Overviews. Structured data is not required for a page to appear in AI Overviews. There is no special "AI Overview" schema type, and anyone selling one is selling a fiction. Schema that matches visible content can still help Google understand entities, products, and FAQs, and that is a real use. Schema that exists only in JSON-LD and contradicts the page is junk, because it describes a page that does not exist when a human opens it. The fiction is the market. A vendor that sells an Overview schema type is selling a thing Google does not have, and the buyer pays for it in setup fees and in the time spent maintaining markup that does nothing.

Promptwatch's schema suggestions are fix hints, not a ranking hack. Use them to fix missing Product or FAQ markup you already show humans, and delete the markup that describes a block you removed. The point of schema here is to describe the page accurately, not to trick a model into a citation. Accurate schema is a description. Inaccurate schema is a lie about the page, and a model that reads the page and finds the schema contradicts it has a reason to distrust the page.

llms.txt without the sales pitch

Promptwatch has an llms.txt generator. Promptwatch's own research, llms.txt has no impact on AI search, finds little or no effect on AI search visibility. Do not buy or build an llms.txt as an Overview lever. Do not claim Google uses the file, because the research and Google's docs do not support it. If you publish one, treat it as optional documentation for agents that choose to read it, the same way a humans.txt file is optional. The generator existing is not a reason to ship the file. The honest position is that the file is harmless and mostly inert, and the time spent on it is time not spent on robots.txt and crawl errors, which are the surfaces that move citations.

Otterly.AI from $29/mo includes a GEO audit of on-page factors. That is a page grade, not crawler logs. Scrunch AI from $250/mo annual serves alternate markup to bots, which is a different bet and one that can read as cloaking if the served version diverges too far from the human version. Semrush AI Toolkit is $99/domain and lives next to classic SEO tech audits. Method: how we rank. Tools list. The three alternatives answer different questions. Otterly grades the page. Scrunch serves a different page to bots, which is a real strategy and a real risk. Semrush puts the audit next to classic SEO work for teams already in the suite.

Who should pick which

Pick Promptwatch Essential if you want crawlability analysis and schema hints on a paid plan with the full engine set. Pick Promptwatch Professional or an agency plan if you want the log side of crawlability joined to the prompt ledger. Pick Otterly if you want a cheap page-grade audit. Pick Scrunch if you have engineering and want to serve alternate markup to bots. Pick the Semrush Toolkit if Semrush is already paid and you want a tech audit next to classic SEO. The picks split by what you need to know. Essential tells you what to change. Professional tells you what happened. Otterly grades the page. Scrunch changes what the bot sees. Semrush adds an audit to a suite you already pay for.

FAQ

Should we add FAQ schema just for AI Overviews?

Only if the FAQ is on the page. Google does not require schema for Overviews and has no Overview-specific type, so schema without matching content is markup that describes a page that does not exist.

Does publishing llms.txt unlock ChatGPT citations?

No. Promptwatch's research says the file has little or no impact. Fix robots.txt and crawl errors first, because those are the surfaces that gate a citation.

What to do this week

Read robots.txt and confirm GPTBot, OAI-SearchBot, PerplexityBot, and ClaudeBot are allowed on public money pages, because a block on any of them is the most common silent miss.
Run crawlability analysis in Promptwatch and fix the first 403 or soft 404 you see.
Match schema to visible text. Delete markup that describes a block you removed, so the schema does not describe a page that no longer exists.
Skip llms.txt unless you want a human-readable map. Do not schedule work around it, because the research does not support an effect.
On Professional, Business, or an agency plan, recheck Agent Analytics after the robots change so you know a bot returned. Essential has no listed crawler-log allowance, so use your own server evidence there.