Agentic Web Optimization: A Brand Guide

An AI agent on your site is doing a job for someone else. It might be fetching a page because a user pasted your URL into ChatGPT, indexing your help center for Claude's search, or reading your product feed so a shopping answer can show a price. None of these visitors look at your hero banner. They read text, follow links, parse feeds and leave.

Agentic web optimization is the work of making a site hold up under that kind of visit. Little of it is new, but it crosses more teams than classic SEO: legal decides about training crawlers, IT runs the firewall, ecommerce owns the feed, and marketing gets asked why the brand is missing from an AI answer. So this checklist puts an owner on every section. If nobody in your company owns an item, that's your first finding.

1. Agent access, split by purpose

Owner: web or platform team for the rules; legal or policy for the training decision.

The vendors don't treat "AI bots" as one category, and neither should you. Each runs separate agents for separate jobs.

Search crawlers are allowed on public pages. OpenAI's OAI-SearchBot surfaces sites in ChatGPT search, and OpenAI's publisher FAQ says to allow it if you want to appear there. Anthropic's Claude-SearchBot indexes pages for Claude's search quality. Perplexity's PerplexityBot surfaces and links sites in Perplexity search and, according to Perplexity, isn't used for training. Blocking these gives up an answer surface for very little.
Someone has written down a decision on training crawlers. GPTBot (OpenAI) and ClaudeBot (Anthropic) collect training data. That's a content licensing call, not an SEO one, and writing it down stops the next CDN migration from quietly reversing it.
Everyone understands Google-Extended before touching it. Google describes it as a robots.txt control token, with no user agent of its own, for Gemini training and grounding. It doesn't affect Google Search inclusion. Because grounding is covered, it isn't a pure training opt-out.
User-triggered fetchers are handled on purpose. ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person asks the assistant to. OpenAI says robots.txt rules may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores them. Claude-User does respect robots.txt, per Anthropic. Stopping the first two takes a firewall rule, and then a customer asking an assistant about your return policy gets nothing from your site.
Rules cover every subdomain. Anthropic notes each of its bots has to be blocked separately per subdomain.
Verification goes beyond the user agent string, which can be faked. OpenAI and Perplexity publish IP lists for their agents, and Perplexity recommends matching user agent and IP together in firewall rules. Google points to its IP ranges and reverse DNS. Anthropic warns IP blocking may not work reliably for its bots.

Give any change a day: OpenAI says search robots changes take about 24 hours, Perplexity up to 24. For schema and crawlability work on the same pages, see our guide to technical optimization for AI crawlers.

2. Facts an agent can read and repeat

Owner: ecommerce or product marketing for products and prices; customer support for policy and contact pages.

Agents compare. If your plan page, your feed and a partner listing give three different prices, the agent picks one of them or drops you.

Every product, plan and price appears in page text, not only in images, click-to-open tabs or a PDF.
Each product has one name everywhere. If the feed says "Pro Bundle" and the page says "Professional Kit," change one.
The return policy lives on its own crawlable URL, with the return window and who pays for return shipping stated plainly.
Support contact details sit on a crawlable page.
Headings describe what's under them ("Shipping to Canada," not "Good to know").
Facts that change carry a date, so an outdated copy is recognizable as outdated.

Google's checkout program in Merchant Center asks for the same things: a defined return policy (return cost, return window, a link to the full policy) and support info with at least one of a website URL, email or phone number. Clean pages turn that setup into a copy job.

3. Product feeds

Owner: ecommerce operations.

Your Google Merchant Center account is in good standing with approved products. Everything Google does in shopping starts there, including the UCP checkout covered below.
You've decided on a ChatGPT product feed. On 24 March 2026 OpenAI extended ACP, the Agentic Commerce Protocol it built with Stripe, to product discovery, so merchants can share feeds and promotions. Shopify merchants are covered through Shopify Catalog. OpenAI's merchant FAQ says a feed isn't strictly needed if ChatGPT already crawls your site, but it gives more control and accuracy.
Feed IDs are stable. The ACP product spec requires a stable id per product and at least one variant with its own stable id and title. Price, media, GTIN and URLs are optional but improve display, so fill them in.
Feed prices match page prices on the same day.

4. Commerce protocols: readiness, not hype

Owner: engineering lead, with payments or your PSP.

Four acronyms come up in vendor pitches. MCP connects AI apps to tools and data; it isn't a checkout protocol. ACP gets products into ChatGPT, and OpenAI says it's moving away from a standalone Instant Checkout, so purchases complete on merchant-owned sites. UCP is Google's open standard for discovery, buying and post-purchase support. AP2 proves a user authorized an agent's purchase.

Most brands don't need to build any of these this quarter. Google's UCP checkout is "available for select merchants at this time," through an early access form, for products eligible in the US, Canada and Australia. Subscriptions, pre-orders, used goods, age-restricted items and services are among the excluded categories.

Ask your PSP whether it's integrated with the Google Pay API. UCP checkout needs a Google Pay & Wallet Console account even if your own site has no Google Pay button.
Check your catalog against Google's ineligible list before anyone scopes work.
Note that native_commerce(checkout_eligibility) defaults to FALSE, so nothing switches on by accident.
File AP2 under "watch." Version 0.2 has been donated to the FIDO Alliance, where standardization continues.

5. Measurement

Owner: marketing or SEO lead.

Robots.txt tells you what you asked for, not what happened. You need to see which agents fetched which pages (and with what status code), which pages get cited, and whether AI-referred visitors convert. ChatGPT's utm_source=chatgpt.com referral tag covers one assistant.

This is where we'd put Promptwatch. Agent Analytics reads AI crawler logs from your CDN (Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN or custom HTTP) and shows which crawlers hit which pages, the errors they got, and the crawl-to-citation path. Citation analytics shows which of your pages get cited and which Reddit threads or other domains are cited instead. Visitor analytics, installed with a script or GTM template, ties AI-referred sessions to conversions. Product brands can track ChatGPT Shopping placements. When the data shows a gap, Content Agents draft the missing page into a review inbox and publish to Webflow, Framer or WordPress once someone approves it.

Questions to ask any vendor before signing

Do you show crawler visits per agent, with status codes, or only prompt mentions?
Can I see which of my pages were cited, not just whether my brand was named?
Can you attribute conversions to AI referrals?
Do you track ChatGPT Shopping placements?

Promptwatch covers each of these. Crawler log volume is listed from the Professional plan at $245/mo (25M crawler logs, 150 prompts, 2 projects). Essential is $95/mo with 50 prompts, 200K visitor events and a 7-day trial. ChatGPT Shopping tracking starts on Business at $579/mo. Annual billing is 12 months for the price of 10. Compare the rest on our tool rankings, or start at promptwatch.com.