Should you block AI crawlers?

"Should I block AI crawlers" is two questions wearing one coat, and they have different answers.

• chevron_rightShould I let AI companies train on my content? A rights and compensation question. You get nothing directly in return, and reasonable people answer no.
• chevron_rightShould I let AI assistants cite my content? A distribution question. Being citable means being recommended to people asking questions in your space.

Because these are served by different user-agents, you do not have to answer them the same way — the mechanics are in GPTBot vs OAI-SearchBot. Almost everyone who thinks about it for more than a minute lands on "no to training, yes to citation", and the main reason more sites are not configured that way is that they copied a block list rather than making a decision.

• chevron_rightYour content has value and training use returns nothing. No payment, no attribution, no traffic. If your business is the content itself, that is a straightforward loss.
• chevron_rightAI answers can substitute for the visit. Someone who gets a sufficient answer in ChatGPT does not click through, even when you are cited — the same dynamic as a featured snippet, more pronounced.
• chevron_rightCrawl load is real on large sites. AI crawlers have grown aggressive enough that some publishers block them for bandwidth reasons alone.
• chevron_rightBlocking is reversible and cheap. Removing a Disallow later costs one line; content already in a training corpus cannot be recalled.

• chevron_rightA growing share of research and comparison queries starts in an assistant rather than a search box, and those users tend to arrive with high intent when they do click.
• chevron_rightCitations compound. Being the source an assistant reaches for on a topic builds a positional advantage that is hard for a competitor to displace.
• chevron_rightBlocking is invisible in your analytics. There is no report showing the answers you were excluded from, so the cost never shows up as a number — which makes it easy to underestimate indefinitely.
• chevron_rightYour competitors' decisions matter. If they stay citable and you do not, the assistant recommends them by default in your category.

News publishers and paid content

The strongest case for blocking training, and often for negotiating a licence instead. Your content is the product and substitution is a direct revenue loss. Many large publishers block training while staying citable, or block everything until a paid agreement is in place.

SaaS, B2B and services

Almost always keep answer engines open. Your content exists to generate demand, not to be sold, and "which tool should I use for X" is exactly the kind of question people now ask an assistant. Being absent from that answer is a marketing decision nobody meant to make.

Ecommerce

Keep answer engines open. Product discovery through assistants is growing and there is little to protect — your descriptions are not the asset, your inventory and prices are. Training blocks are largely symbolic here.

Independent blogs and creators

The genuinely hard case, and the one where the answer is personal. If your writing is the point, blocking training is a defensible stance on principle. If discovery is the constraint, citation is one of the few distribution channels not already saturated. Splitting the decision by agent is the compromise that costs least.

Documentation and open-source projects

Allow both, usually. Being the source an assistant quotes when someone asks how to use your library is unambiguously good for adoption, and it reduces the volume of questions that land on your issue tracker.

Both choices can be changed by editing one file, but the consequences unwind at very different speeds, and that asymmetry should weigh on the decision more than it usually does.

Training: blocking late is worth little

Content already collected is already in a corpus. Adding a Disallow stops future collection; it does not withdraw what was taken, and there is no mechanism to request removal from an already-trained model. If you are going to block training, blocking early is the only version that does much — which is the strongest practical argument for the training block, independent of how you feel about the technology.

Citation: unblocking is slow but complete

Removing a Disallow on an answer engine restores eligibility fully, but not instantly — the engine has to recrawl, and there is no published turnaround. Treat it as weeks. The corollary is that the cost of having been blocked is not recovered either: the months you were absent from answers are simply gone, and any positional advantage a competitor built in that time stays theirs.

Several proposals are circulating for giving publishers finer control than robots.txt offers — separating training from inference, or attaching licensing terms to content. None is settled, and none is a substitute for robots.txt today.

• chevron_rightllms.txt is a proposed file describing your site's structure for AI systems. It helps them understand you; it is not an access-control mechanism and no major engine has committed to it.
• chevron_rightThe IETF's AI preferences work aims at a standard vocabulary for stating training permissions, but it is in progress and unimplemented.
• chevron_rightVendor-specific tokens like Google-Extended and Applebot-Extended are the closest thing to granular control that actually works right now — and only for those vendors.

So the practical answer is unchanged: robots.txt is the control surface, agent by agent. Adding an llms.txt is worthwhile for readability, not for permission — see generative engine optimization for what it is actually good for.

Partly, and the gaps matter — because a decision you cannot measure tends to drift toward whichever side is louder.

What you can see

• chevron_rightReferral traffic from chatgpt.com, perplexity.ai, claude.ai and copilot.microsoft.com appears in your analytics referrer report as normal.
• chevron_rightThat traffic tends to convert well, because the assistant has already filtered for relevance before the person clicks.
• chevron_rightYour logs show which answer engines crawl you and how often, which is a rough proxy for whether you are in the index at all.

What you cannot see

• chevron_rightImpressions. There is no Search Console equivalent for AI answers — no report of how often you were cited without a click.
• chevron_rightAnswers you were excluded from. A citation that never happened leaves no trace anywhere.
• chevron_rightAttribution when the assistant summarises without linking, which is common for factual queries.
• chevron_rightWhether a competitor is being cited in your place.

It is not binary. Three intermediate positions are worth knowing before you settle on all-or-nothing.

Split by agent

Block training, allow answering. The default recommendation, and the one most publishers reach independently once the distinction is clear.

Split by path

robots.txt takes path rules per agent, so you can expose some sections and protect others. A company might let answer engines read its documentation and blog — the parts that generate demand — while keeping training crawlers out of an archive of paid research.

  User-agent: GPTBot
  Disallow: /research/
  Disallow: /premium/
  Allow: /blog/

  User-agent: OAI-SearchBot
  Allow: /

Rate-limit instead of block

If your objection is crawl load rather than principle, `Crawl-delay` is honoured by some agents, and CDN-level rate limiting works on all of them. That keeps you citable while capping the bandwidth cost — a better answer than a block when bandwidth was the real complaint.

The worst outcome is not blocking or allowing — it is not knowing which you did. That happens more than it should, because robots.txt accumulates: a staging rule, a security plugin's defaults, a snippet from a blog post, a CDN rule nobody documented.

1. Fetch your /robots.txt and read it as it stands today, not as you remember writing it.
2. Check for a wildcard `Disallow: /` first — it overrides everything you thought you configured.
3. List which AI agents are named, and confirm each one is on the side you intended.
4. Add a comment above each block explaining the intent, so the next edit does not undo the decision.
5. Re-check after any security-plugin or CDN change — both add bot rules without announcing it.

RankForge's free AI Crawler Checker does step 2 and 3 for you, resolving precedence per agent and separating answer engines from training crawlers. It reports what your file does and never scores you down for either choice — the decision is yours to make, not ours to grade.

This decision only works if you know what you are allowing. New agents appear regularly and vendors split a single crawler into training and answering agents once publishers start blocking the original, so a block list written two years ago is very likely aimed at the wrong half.

The AI crawler user-agent reference lists every major agent, which column it belongs in, and what blocking each one actually costs — including the ambiguous ones like bingbot, where blocking removes you from a search engine rather than from AI.