GPTBot vs OAI-SearchBot: which one should you actually block?
There is a line in a lot of robots.txt files that was meant to say "don't train on my content" and actually says "don't recommend me to anyone". It usually appears after someone reads a post about blocking AI, copies a block of user-agents, and pastes it in. The problem is that the AI crawlers hitting your site are not doing the same job as each other, and the two OpenAI agents most often blocked together — GPTBot and OAI-SearchBot — have opposite consequences. One collects content for model training. The other fetches pages so ChatGPT can cite them in an answer. This guide explains the difference, what blocking each actually costs, and how to configure robots.txt to make that choice on purpose.
Run the AI Crawler Checker on your site — free, no account.
The three OpenAI agents, and what each does
OpenAI publishes three separate user-agents, and they exist as separate agents specifically so publishers can treat them differently.
GPTBot — training
GPTBot crawls the open web to gather content that may be used to train future models. Nothing it fetches is shown to a user in real time, and nothing it fetches sends you a visitor. Blocking it is the "do not train on my work" decision, and it is the one most publishers who block anything mean to make.
OAI-SearchBot — the search index
OAI-SearchBot builds the index that ChatGPT's search feature draws on. When ChatGPT answers a question with links to sources, those sources come from this index. Blocking OAI-SearchBot removes you from that index — functionally the same effect as putting a noindex on your pages, but for AI search rather than Google. It does not protect your content from training; that is a different agent entirely.
ChatGPT-User — live, user-initiated fetches
ChatGPT-User fires when a person in a conversation asks about your page specifically, or follows a link. It is not a bulk crawler — it is one fetch, prompted by one human, right now. Blocking it means a user who explicitly asks ChatGPT to read your page gets told it cannot.
agent fetches for blocking it costs you -------------- -------------------- --------------------------- GPTBot model training nothing you can measure OAI-SearchBot ChatGPT search index citations, referral traffic ChatGPT-User one live user request that specific user, right now
Why they get blocked together
The confusion has a simple cause: GPTBot launched first and got all the coverage. Through 2023 and into 2024, "how to block AI crawlers" meant "block GPTBot", and thousands of robots.txt files were written against that advice. OAI-SearchBot arrived later, and the snippets circulating online were updated by people adding it to the same block list — reasonably enough, since it has OpenAI's name on it.
The result is a lot of sites that opted out of AI citations while believing they opted out of AI training. It is worth checking rather than assuming, because the cost is invisible: there is no report anywhere that tells you how many AI answers you were excluded from.
The wildcard trap A `User-agent: *` with `Disallow: /` blocks every AI agent as a side effect of blocking everything. If that rule survived from a staging environment, it is also keeping you out of Google — check for it first, before anything AI-specific.
The same split at other vendors
OpenAI is not unusual here. Most major AI vendors now separate training from answering, and the same reasoning applies to each:
- Anthropic — ClaudeBot crawls for training; Claude-User and Claude-SearchBot support answering and citation.
- Perplexity — PerplexityBot builds the index; Perplexity-User fetches on a live request.
- Google — Googlebot is search; Google-Extended is a robots.txt token controlling Gemini training. Blocking Google-Extended has no effect on Google Search ranking at all.
- Apple — Applebot serves Siri and Spotlight; Applebot-Extended is the training opt-out token.
Tokens are not crawlers Google-Extended and Applebot-Extended never fetch anything. They are directives that control how content already fetched by Googlebot or Applebot may be used. You cannot see them in your server logs, and blocking them costs you no crawling or ranking — which makes them the cleanest training opt-out available.
The exception worth knowing is Bing. Bingbot powers both Bing search and Microsoft Copilot's grounding, and there is no separate token. Blocking bingbot to keep out of Copilot also removes you from Bing search — so it is not a clean AI opt-out, it is a search opt-out.
Checking your logs to see which agents actually visit
robots.txt tells you what you permit. Your access logs tell you what is actually happening, and the two diverge more often than people expect — an agent you never blocked may simply not be crawling you, which is a different problem with a different fix.
Grep your access log for the agent tokens. On a typical nginx or Apache setup:
AGENTS='GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot' AGENTS="$AGENTS|Claude-User|PerplexityBot|CCBot" grep -ohE "$AGENTS" access.log | sort | uniq -c | sort -rn # 482 GPTBot # 119 OAI-SearchBot # 37 PerplexityBot # 4 ChatGPT-User
- No AI agents at all, and nothing blocked — you are probably too new or too small to be crawled yet, not misconfigured.
- GPTBot present, OAI-SearchBot absent — normal. The search index crawls far less aggressively than the training crawler.
- Everything absent while robots.txt looks open — check for a CDN or WAF rule blocking bots above the application layer, where robots.txt has no say.
- ChatGPT-User hits — someone is asking ChatGPT about your pages by name. Small numbers, high intent.
User-agents are self-declared Anyone can send any user-agent string, so a log line is a claim rather than proof. OpenAI, Anthropic and Perplexity all publish IP ranges for their crawlers — verify against those before drawing conclusions about traffic volume or building blocks around them.
What getting it wrong looks like
Blocking the answer engine by mistake
A B2B software company adds the popular AI block list to robots.txt in 2024, believing it protects their documentation from training. The list includes OAI-SearchBot. For the next year, every time someone asks ChatGPT to recommend a tool in their category, competitors are cited and they are not — because the index cannot fetch their pages. Nothing in their analytics changes visibly, because traffic that never happened does not appear in a report. This is the common failure, and it is silent by construction.
Blocking nothing when you meant to block training
The reverse is louder but usually less costly. A publisher intends to opt out of training, adds `User-agent: GPTBot`, and stops there — leaving ClaudeBot, CCBot and Google-Extended untouched. Because CCBot's dataset feeds many models, the opt-out achieves considerably less than intended. The fix is a complete list rather than a single line.
Both failures come from the same root: treating "AI crawlers" as one thing. Once you list agents by job rather than by vendor, the right configuration is obvious.
Configuring it deliberately
If your intent is "opt out of training, stay citable in AI search" — which is what most publishers want once the distinction is clear — the shape is straightforward. Name the training agents, disallow them, and leave the answer agents alone:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / # OAI-SearchBot, ChatGPT-User, PerplexityBot, # Claude-User: intentionally not listed, so # they inherit the default and can cite us.
- Agent names are matched case-insensitively, so GPTBot and gptbot behave identically.
- A named group wins over the wildcard group. If `*` disallows everything, an agent needs its own `Allow: /` group to get through.
- Group your rules with a blank line between agents. Consecutive `User-agent:` lines with no rule between them share the block that follows.
- robots.txt is a request, not enforcement. The agents above publish their user-agents and honour it; anything that ignores it has to be stopped at your CDN or firewall.
Being crawlable is not the same as being cited
Unblocking OAI-SearchBot makes you eligible for citation. It does not make you cited, and it is worth being clear about that before treating a robots.txt edit as a growth strategy.
An answer engine picks sources the way a careful researcher would: it needs the page to load, to contain the answer in readable text, and to be recognisably about the thing asked. Two of those are structural. A page whose content only appears after JavaScript runs is fetched successfully and read as nearly empty — the crawler got in and found nothing to quote. A page with no clear topical context around it is harder to trust than one sitting inside an obvious cluster on the subject.
- Server-render the content that answers the question. Client-side rendering is the single most common reason an allowed page is never cited.
- Answer the question near the top, in prose. Assistants quote passages, not page structures.
- Keep the page inside a topical cluster — supporting pages linking to a clear hub make the subject legible.
- Give the page internal links. A page nothing links to reads as unimportant to an assistant for the same reason it does to a search engine.
That last point is the overlap with ordinary structural SEO: how authority flows through your site decides which of your pages look like the canonical answer, and that judgement is not made differently by an assistant than by a search engine.
Checking what you currently do
Reading your own robots.txt is harder than it sounds once wildcard groups, named groups and Allow overrides interact — which agent wins is a resolution problem, not a reading problem. RankForge's free AI Crawler Checker resolves the rules per agent and reports answer engines and training crawlers separately, so you can see which choice you actually made.
Crawler access is only the door, though. Being allowed in does not mean being cited — AI engines still have to be able to read the page once they arrive, which is where server-rendered content and clear structure matter. Generative engine optimization covers what happens after the fetch succeeds.
FAQ
Does blocking GPTBot stop ChatGPT from citing my site?
No. GPTBot only collects content for model training. ChatGPT's citations come from the index built by OAI-SearchBot, and from live fetches by ChatGPT-User. You can block GPTBot and remain fully citable in ChatGPT search — many publishers deliberately do exactly that.
Will blocking Google-Extended hurt my Google rankings?
No. Google-Extended is a robots.txt token that controls whether your content may be used for Gemini training and grounding. It is not a crawler and has no role in Google Search indexing or ranking. Blocking it is the lowest-risk training opt-out available.
Should I block ChatGPT-User?
Usually not. ChatGPT-User only fires when a person explicitly asks ChatGPT about your page or follows a link to it. Blocking it means turning away a visitor who specifically asked for your content — closer to blocking a browser than blocking a crawler.
Is there one line that blocks all AI training?
No, because the agents belong to different companies and each must be named. There is also no shared standard yet — llms.txt is an emerging proposal, not an opt-out mechanism. Practically you list the training agents you care about (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) and leave the answer engines alone.
How do I know if my robots.txt is already blocking the wrong thing?
Fetch /robots.txt and look for any named AI agent, plus any `User-agent: *` with `Disallow: /`. Resolving precedence by hand is error-prone once wildcard and named groups overlap, so a checker that reports the outcome per agent is faster and less likely to miss an override.
Will allowing OAI-SearchBot get me traffic from ChatGPT?
It makes you eligible, which is not the same as guaranteed. The engine still has to judge your page a good source: the content has to be in the server-rendered HTML rather than loaded by JavaScript, it has to answer the question in readable prose, and the page has to look important enough within your own site to be worth quoting. Access is necessary and not sufficient.
Can I allow GPTBot on some pages and block it on others?
Yes. robots.txt takes path rules per agent, so you can disallow GPTBot from a paid archive while allowing it on your blog. Put the Disallow lines for the specific paths under the GPTBot group; they apply only to that agent.
How do I know if the GPTBot in my logs is real?
Check the source IP against the ranges OpenAI publishes for each of its agents. User-agent strings are self-declared and scrapers routinely impersonate well-known crawlers, so a log line on its own proves nothing about who actually made the request.
Put this into practice
Run the AI Crawler Checker to see this on your own site, or run the full structural audit for the complete picture — both free, no account required.