OpenAI Search Crawler OAI-SearchBot Documentation

OpenAI search crawler OAI-SearchBot documentation lives on Overview of OpenAI Crawlers. OAI-SearchBot is for search. It surfaces websites in ChatGPT's search features. Sites opted out will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from published IP ranges (searchbot.json). The recommendation is two parts, and both parts have to be in place. The robots.txt allow tells the bot it may fetch. The IP range allow at the CDN lets the fetch arrive. A team that does the first and skips the second has a robots file that says yes and an edge that says no, and the citation program is blocked at the edge without an error the content team can see.

This post stays with those official pages. It is not a GEO feature list dressed up as crawler policy. The reason for that line is that most "GEO" posts about this bot spend two sentences on the bot and five on a vendor dashboard. We invert that. The bot policy is the work. The dashboard is the readout. A dashboard that logs a fetch the bot never made is a readout of a block, not a visibility win, and a team that reads it as a win reads the wrong column.

Example user-agent, with the caveat that the version may change: compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. Robots.txt fetches may include an extra robots.txt marker in the UA. If you grep logs, search for that pattern and for the published IP ranges. A robots allow with a CDN deny still fails, because the CDN sits in front of the robots file. Version numbers change, and the docs are the source, not a blog memory of 1.4. A grep that hard-codes 1.4 stops matching the day OpenAI ships 1.5, and the team that wrote the grep believes the bot stopped fetching when the bot only changed its version string.

Do not mix the bots

GPTBot is training data. ChatGPT-User is used when a user, or a Custom GPT, fetches a page. Robots.txt may not apply the same way to User, and it is not the Search opt-out control. Use OAI-SearchBot in robots.txt to manage Search. The three user agents are separate, and the policies are separate. A team that writes one rule for all three writes a rule that does three different things, and the team cannot tell which of the three it controlled.

Three decisions sit in front of you, and they are independent. Search is OAI-SearchBot. Training is GPTBot. User-initiated fetches are ChatGPT-User. You can block GPTBot and allow OAI-SearchBot. Those are different decisions, and conflating them is the most common error in a GEO ticket. Document both. Do not use a single "Disallow / OpenAI" rule and call it a Search policy, because that rule blocks the search surface you are trying to measure. The single broad rule is the part that causes the silent drop. Legal writes it, the team copies it, and the next QBR shows a citation drop nobody can trace back to the robots file.

The publishers FAQ repeats the cost of getting this wrong. Block OAI-SearchBot and you hurt summaries and snippets. utm_source=chatgpt.com tags clicks when they happen, so the referral column only populates if the search bot was allowed in the first place. The tag and the allow are linked. A team that turns on the referral tracking without the allow sees an empty referral column and blames the tracking, when the empty column is the symptom of the block.

CDN "bot protection" presets often block these by default. Check logs, not only robots.txt intent. Allowing the published IP ranges at the CDN is the other half of the official recommendation, and it is the half that gets missed when the ticket is owned by content and not by infra. The preset is the part that looks like a policy and acts like a block. A team that turns on a preset labeled "block AI bots" believes it only blocked training. The log shows whether it blocked search too, and the log is the part that tells the truth.

After you have allowed the bot

Promptwatch stores the ChatGPT Search answer for prompts on its paid plans. Essential is $95/mo, but it does not include crawler logs. Professional at $245/mo adds Agent Analytics with 25M crawler logs, crawl-to-citation paths, and error tracking. Explore is limited to free ChatGPT prompts. Site: promptwatch.com. Method: how we rank. The plan split is the part that decides what you can log. Essential stores the answer. Professional stores the answer and the crawl. A client who needs proof the fetch happened needs Professional or an agency plan, because the fetch row is the row that settles the "did the bot read us" question.

Allowing the bot does not guarantee a citation. It removes a self-inflicted 403.

That is the only promise. You unblock Search. You still have to earn the mention. On Professional, Agent Analytics tells you whether the fetch happened and whether it errored. Prompt tracking on paid plans tells you whether the Search answer named you. Explore and Essential do not include crawler logs, so a team on those plans can read the answer but not the fetch row. The two questions, did the bot fetch, and did the answer name us, are different questions. The fetch is a webmaster question. The name is a content question. A team that fixes the fetch and stops will unblock the bot and still not be cited, because the content question was never answered.

Load the prompts those pages should win after the robots change. A bot allow without a prompt list is a webmaster ticket, not a visibility program, and the visibility program is the thing the client is paying for. The prompt list is the part that turns the allow into a program. Without the list, the team has unblocked the bot and has no way to know whether the unblock changed anything.

FAQ

Is OAI-SearchBot the same as GPTBot?

No. Search versus training. They are separate crawlers with separate opt-out controls. One feeds ChatGPT Search answers. The other feeds model training.

Can I block GPTBot and allow OAI-SearchBot?

Yes. Those are different decisions. Document both. Read the bots page and keep the user-agent strings in the runbook.

What to do this week

1. Read the bots page. Then the publishers FAQ. The docs are the source. A vendor paraphrase is not.
2. Allow OAI-SearchBot on pages you want in ChatGPT Search. The allow is the first half of the recommendation.
3. Allow the published IP ranges at the CDN (searchbot.json). The IP allow is the second half. The first half without the second half still fails.
4. Connect crawler logs in Promptwatch Professional, which includes a 25M allowance. Essential does not include crawler logs.
5. Load the prompts those pages should win. The prompts are the part that turns the unblock into a program.

Official OpenAI documentation is the source for OAI-SearchBot, GPTBot, and ChatGPT-User. A GEO dashboard can log the fetch. It cannot rewrite the bot policy. Keep the docs in the ticket, not a vendor paraphrase.