AI crawler user-agents: the full reference list
A reference list of the AI crawlers currently fetching pages from the open web, grouped by what they are for rather than by who operates them — because the vendor matters far less than the job. Each entry notes what the agent does, whether blocking it costs you anything visible, and any quirks worth knowing. The critical division runs between crawlers that collect content for model training and agents that fetch pages so an assistant can cite them; blocking the first is an editorial choice, blocking the second removes you from AI answers.
Run the AI Crawler Checker on your site — free, no account.
AI answer engines
These fetch pages so an assistant can reference them in a response. Blocking one is functionally a noindex for that engine.
- OAI-SearchBot (OpenAI) — builds the index behind ChatGPT search. Block it and you cannot be cited in ChatGPT.
- ChatGPT-User (OpenAI) — a live fetch triggered by a person asking about a specific page. Not a bulk crawler.
- Claude-SearchBot (Anthropic) — indexing for Claude's search results.
- Claude-User (Anthropic) — live, user-initiated fetch within a Claude conversation.
- PerplexityBot (Perplexity) — builds Perplexity's index. Perplexity cites heavily, so this one carries unusually direct referral value.
- Perplexity-User (Perplexity) — live fetch on a user request.
- DuckAssistBot (DuckDuckGo) — powers DuckDuckGo's AI assist answers.
- YouBot (You.com) — indexes for You.com's assistant.
- MistralAI-User (Mistral) — live fetch from Le Chat.
- Amazonbot (Amazon) — feeds Alexa answers as well as Amazon search surfaces.
- Applebot (Apple) — serves Siri and Spotlight suggestions. Also the crawler whose output Applebot-Extended governs.
AI training crawlers
These collect content for model training corpora. They send no traffic back, so blocking them costs nothing you can measure — which is why blocking training is the low-risk half of the decision.
- GPTBot (OpenAI) — the original and most-blocked AI crawler. Training only; unrelated to ChatGPT citations.
- ClaudeBot (Anthropic) — Anthropic's primary crawler.
- anthropic-ai — a legacy agent string still seen in older robots.txt files; harmless to keep listed.
- CCBot (Common Crawl) — builds the public Common Crawl dataset, which many models train on. Blocking it affects multiple downstream trainers at once.
- Google-Extended — a robots.txt token, not a crawler. Controls Gemini training and grounding use. No effect on Google Search.
- Applebot-Extended — likewise a token, governing whether Applebot's crawl may be used for Apple's model training.
- Meta-ExternalAgent (Meta) — crawls for Meta AI training.
- Bytespider (ByteDance) — widely reported as aggressive; often blocked for crawl-load reasons rather than principle.
- cohere-ai (Cohere) — training data collection.
- AI2Bot (Allen Institute) — collects for open research datasets.
- Omgilibot / Webzio-Extended (Webz.io) — feeds a commercial dataset resold for training.
- Diffbot — builds a structured knowledge graph sold to AI and data customers.
Tokens vs crawlers Google-Extended and Applebot-Extended never make a request. They are directives controlling how content already fetched by Googlebot or Applebot may be used, so you will never see them in server logs and blocking them costs no crawl coverage at all.
Agents that are not cleanly either
- bingbot (Microsoft) — powers Bing search and grounds Microsoft Copilot, with no separate token. Blocking it to avoid Copilot also removes you from Bing search.
- Googlebot (Google) — powers Search and, through Google-Extended's separate control, feeds AI Overviews. Never block it as an AI measure; use Google-Extended instead.
- facebookexternalhit (Meta) — generates link previews when someone shares your URL. Blocking it breaks social previews and has nothing to do with AI training.
- Google-CloudVertexBot — fetches sites on behalf of Vertex AI customers building their own agents, rather than for Google's own training.
These are where copied block lists do the most damage. A list that lumps bingbot or facebookexternalhit in with GPTBot costs you search visibility and social previews for no AI benefit.
Verifying an agent is genuine
A user-agent string is self-declared, so any scraper can call itself GPTBot — and plenty do, precisely because sites that want AI traffic will let it through. Before you act on log volume, verify.
- OpenAI publishes JSON files of the IP ranges used by GPTBot, OAI-SearchBot and ChatGPT-User separately, updated as they change.
- Anthropic publishes IP ranges for ClaudeBot and its user-initiated agents.
- Perplexity publishes ranges for PerplexityBot.
- Google and Apple support reverse DNS verification — resolve the IP to a hostname, then resolve that hostname back and confirm it matches.
The practical rule: verify by IP before blocking on user-agent, because a block written against a spoofed string does nothing to the spoofer and everything to the genuine crawler.
When robots.txt is not enough
robots.txt is a request. Well-behaved agents honour it; a scraper harvesting content for a dataset has no reason to. If your objection to AI crawling is strong enough to matter commercially, enforcement lives a layer down.
- CDN bot rules — Cloudflare, Fastly and Akamai all ship managed AI-bot categories that block at the edge, before a request reaches your server.
- WAF rules by user-agent or IP range — blunter, and worth pairing with the published ranges above so you do not block the real crawler while missing the fake one.
- Authentication — content behind a login is not reachable by any crawler, which is the only genuinely reliable protection.
- Rate limiting — the right answer when the complaint is bandwidth rather than principle, since it keeps you citable.
Check the layer above This cuts both ways. A managed CDN bot rule can block agents your robots.txt happily allows, so a site that looks open can be closed in practice. If your logs show no AI agents at all despite an open robots.txt, check your CDN settings before assuming you are simply not being crawled.
How the matching actually works
- Agent names match case-insensitively — GPTBot, gptbot and GPTBOT are the same rule.
- Matching is on a substring of the full user-agent string, so you list the token (GPTBot), not the whole header.
- A group naming an agent takes precedence over the `*` group entirely — the wildcard's rules are not merged in, they are replaced.
- Consecutive `User-agent:` lines with no rule between them share the block that follows. A blank line does not end a group; a directive followed by a new `User-agent:` does.
- An agent with no named group falls through to `*`. If `*` disallows the root, that agent is blocked.
That precedence rule is where most robots.txt files go wrong in practice. A file with `User-agent: *` / `Disallow: /` at the top and carefully written AI groups below it blocks everything anyway unless each of those groups carries its own explicit `Allow: /`.
Agentic browsers: the category robots.txt handles badly
A newer class of traffic does not fit either column. Agentic browsing tools — assistants that operate a real browser to complete a task, plus browser extensions and integrated AI sidebars — fetch pages on behalf of a specific person, in real time, often through the user's own session.
Some identify themselves (ChatGPT-User, Perplexity-User, Claude-User are the declared versions of this). Others browse through a genuine browser profile and are indistinguishable from a human visitor, because functionally that is what they are: a person delegating a click.
- Blocking the declared agents does not stop the undeclared ones — it just turns away the traffic honest enough to announce itself.
- These are not training crawlers. Nothing they fetch goes into a corpus; they are answering one person's question right now.
- They arrive with intent. A user who asked an assistant to check your pricing page is closer to a lead than to a bot.
- Aggressive bot-detection at the CDN catches them alongside scrapers, which is a common accidental block.
The useful mental model: treat declared user-initiated agents as browsers rather than crawlers. If you would not block a visitor for using an unusual browser, the case for blocking these is weak.
Where each vendor documents its agents
This list will drift, so the authoritative source is always the vendor. Each of the major operators publishes its agent names and, in most cases, the IP ranges to verify them against:
- OpenAI — publishes GPTBot, OAI-SearchBot and ChatGPT-User separately, each with its own IP range file.
- Anthropic — documents ClaudeBot and the user-initiated agents, with published ranges.
- Google — documents Googlebot, Google-Extended and Google-CloudVertexBot in its crawler reference, and supports reverse DNS verification.
- Apple — documents Applebot and Applebot-Extended, and supports reverse DNS verification.
- Perplexity — documents PerplexityBot and Perplexity-User with published ranges.
- Common Crawl — documents CCBot and its crawl schedule.
Re-check these after any significant AI product launch. The pattern so far has been that vendors split a single crawler into training and answering agents once publishers start blocking the original — which means the block you wrote last year may now be pointed at the wrong half.
Mistakes in circulating block lists
Most AI blocks in the wild were pasted from a blog post rather than written. These are the errors worth checking your own file for:
- Answer engines mixed in with training crawlers — OAI-SearchBot and PerplexityBot appearing in a list labelled "stop AI training". The most common and most expensive mistake.
- bingbot on the list — costs you Bing search entirely, for no AI-specific benefit.
- facebookexternalhit on the list — breaks link previews when anyone shares your URL; nothing to do with AI training.
- Named groups placed below a `User-agent: *` / `Disallow: /` — the wildcard already blocks everything, so the careful groups underneath do nothing without their own `Allow: /`.
- Deprecated agents only — a file listing anthropic-ai and Claude-Web but not ClaudeBot is blocking strings that are no longer the primary crawler.
- No comments — six months later nobody remembers whether a line was deliberate, so it gets copied forward forever.
A starting template
If your intent is the common one — opt out of training, stay citable in AI answers — this is the shape. Adjust the training list to taste; the point is that the answer engines are deliberately absent and the file says so.
# AI training crawlers - blocked deliberately. # Do NOT add OAI-SearchBot, PerplexityBot, Claude-User # or ChatGPT-User here: those are answer engines, and # blocking them removes us from AI citations. User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: Meta-ExternalAgent Disallow: / # Everything else, including all answer engines. User-agent: * Disallow: /wp-admin/ Sitemap: https://example.com/sitemap.xml
Applebot-Extended and Google-Extended are the safest entries here: they are tokens rather than crawlers, so blocking them costs no crawl coverage in Google Search or Siri at all.
The same list, machine-readable
Every agent above is also published as JSON at rankforge.cc/ai-crawlers.json — the same data this page is written from, so the two cannot drift apart. It is served with permissive CORS, so you can fetch it directly from a script or a page.
- agent — the user-agent token to match in robots.txt, lower-cased.
- kind — answer, training, or ambiguous.
- description — what it fetches for, and what blocking it costs.
- token_only — true for Google-Extended and Applebot-Extended, which never make a request and so will never appear in your logs.
- version — a date string that changes whenever an agent is added, removed or reclassified.
Useful if you generate robots.txt from a template, or want to alert on a deploy that starts blocking an answer engine. Check the version field rather than diffing the whole document.
Free to use with attribution (CC BY 4.0). If you spot a missing or misclassified agent, tell us and it will be corrected in the published data as well as here.
Keeping the list current
New agents appear regularly and existing ones get renamed — OpenAI's split into three agents happened after most robots.txt files were written, and Anthropic's search agents arrived later still. A list pasted once and never revisited drifts out of date within months, usually in the direction of blocking things that no longer exist while missing the ones that do.
Rather than auditing this by hand, RankForge's free AI Crawler Checker resolves your robots.txt against the current list and reports answer engines and training crawlers separately. For the decision itself, should you block AI crawlers sets out the trade-off by site type, and GPTBot vs OAI-SearchBot covers the distinction that causes most misconfigurations.
Access is only the first step
Allowing a crawler makes you eligible to be cited; it does not make you cited. The engine still has to fetch the page, find the answer in readable server-rendered HTML, and judge you an authority on the subject.
Generative engine optimization covers what happens after the fetch succeeds, and why internal links matter explains why a page nothing links to is a poor citation candidate for the same reasons it is a poor ranking candidate.
FAQ
What is the difference between GPTBot and ChatGPT-User?
GPTBot is a bulk crawler collecting content for model training. ChatGPT-User is a single fetch triggered when a person in a conversation asks ChatGPT to read a specific page. One is a corpus builder, the other is closer to a browser acting for a user.
Is Google-Extended a crawler?
No. It is a robots.txt token that controls whether content Googlebot already fetched may be used for Gemini training and grounding. It never makes a request, so it will not appear in your server logs, and blocking it has no effect on Google Search crawling or ranking.
Which AI crawlers should I never block?
Googlebot and bingbot, because both are search crawlers first — blocking either as an AI measure removes you from a search engine. Use Google-Extended for Gemini training instead. Bing has no equivalent token, so there is currently no way to opt out of Copilot grounding without leaving Bing search.
How often does this list change?
Several times a year. Vendors add agents as they separate training from answering — OpenAI's three-agent split and Anthropic's search agents both arrived after the first wave of AI-blocking advice was written — so a robots.txt written in 2023 is very likely out of date now.
Do I need to list every agent to block training?
You need to name each one you want blocked, since there is no shared opt-out token across vendors. Blocking CCBot has the widest reach because Common Crawl's dataset feeds many models, but it does not cover crawlers that collect independently, such as GPTBot or ClaudeBot.
What is the difference between ClaudeBot and Claude-User?
ClaudeBot is a bulk crawler collecting content for training. Claude-User is a live fetch made when someone in a conversation asks Claude to read a specific page. Blocking the first is a training opt-out; blocking the second turns away a person who explicitly asked for your page.
Is Bytespider safe to block?
It carries no search or citation traffic you would miss in most markets, and it is widely reported as crawling aggressively, so blocking it is low-risk. The exception is if ByteDance properties matter to your audience — check your referrer data before assuming it is pure cost.
Do I need to block CCBot separately from GPTBot?
Yes. Common Crawl builds a public dataset that many organisations train on, so it is a distinct collection route rather than a duplicate of any one vendor's crawler. Blocking CCBot has unusually wide reach for a single line, but it does not cover crawlers that collect independently.
Sources
Overview of Google crawlers — Google Search Central
OpenAI bots — OpenAI
Put this into practice
Run the AI Crawler Checker to see this on your own site, or run the full structural audit for the complete picture — both free, no account required.