Is your robots.txt blocking AI search?
ChatGPT, Perplexity and Claude cite pages they can fetch. A single line in robots.txt — often copied from a template or left over from staging — can remove you from those answers without anyone noticing. Paste your URL to see which AI crawlers you allow and which you block. Free, no account required.
What does the ai crawler checker show you?
An example of what the ai crawler checker produces — your real result will use your site's data.
AI answer engines
AI training crawlers
How does the ai crawler checker work?
From a URL to an answer in three steps — no setup, no spreadsheets.
Paste your URL
No account, no install, no plugin. Enter any URL and start — RankForge works the way a search engine does.
We crawl & map your site
RankForge crawls up to 100 pages, following internal links and building a map of how your pages connect.
Get your result
See the findings in minutes — with a plain-English explanation and the specific fixes a free account away.
What does this tool check?
What problems does this find?
The issues this check most often surfaces on real sites:
- A blanket Disallow: / left over from staging that blocks every AI engine
- PerplexityBot or OAI-SearchBot disallowed by a copied robots.txt template
- A security plugin or CDN rule blocking AI user-agents without anyone deciding to
What do I do after the check is complete?
We checked whether the AI tools people now ask questions in — ChatGPT, Perplexity, Claude — are allowed to read your website. Some of them read your pages so they can recommend you in an answer; others just collect text to train on. Blocking the second group is fine and normal. Blocking the first group quietly keeps you out of AI recommendations, and it is usually an accident.
- 1
Start with the pages that matter commercially
Work down the list in order of how much you care about the page, not how bad the number looks. A flagged page nobody needs to find is not worth an hour of your time.
- 2
Fix the cause, not the symptom
Most findings here are structural: a page is unreachable, under-linked, or competing with another page. Adding one contextual link from a strong, relevant page usually fixes more than editing the page itself.
- 3
Re-run this check to confirm the fix landed
The check is free and takes a couple of minutes. Re-crawling after a change is the only way to know it worked — and the full audit tracks the difference between crawls for you.
Why is chatgpt blocked from my site matters for SEO
AI assistants answer more and more queries directly, and they cite the pages they were able to fetch. That fetch is governed by robots.txt — and the agents involved are not the ones most robots.txt files were written for. The important distinction is intent: training crawlers such as GPTBot and CCBot collect content for model training, and blocking them is a legitimate editorial decision many publishers make deliberately. Answer engines such as OAI-SearchBot and PerplexityBot are different — they fetch a page in order to cite it, so blocking them removes you from AI answers the way a noindex removes you from Google. The two get confused constantly, and a robots.txt copied from a blog post about “blocking AI” often blocks both. This check separates them so you can make each call on purpose.
More free SEO tools
Learn how to fix it
Guides from the RankForge Academy on what this check measures and what to do about it.
- AI crawler user-agents: the full reference list11 min read
- Should you block AI crawlers?10 min read
- GPTBot vs OAI-SearchBot: which one should you actually block?9 min read
AI Crawler Checker: frequently asked questions
Does blocking GPTBot stop ChatGPT citing my site?
No — they are different crawlers. GPTBot collects content for model training. OAI-SearchBot and ChatGPT-User fetch pages so ChatGPT can cite them in an answer. You can block training while staying citable in AI search; many publishers do exactly that. This check reports the two separately for that reason.
Should I allow AI crawlers?
That is your call, and there is a real trade-off. Allowing answer engines makes you eligible for AI citations and the referral traffic they carry. Allowing training crawlers contributes your content to model training with no direct traffic back. RankForge reports what your robots.txt currently does and never scores you down for either choice.
Does robots.txt actually stop AI crawlers?
For the major, well-behaved agents listed here — OpenAI, Anthropic, Perplexity, Google, Apple — yes, they publish their user-agent strings and respect robots.txt. It is not an enforcement mechanism, though: robots.txt is a request, and crawlers that ignore it need blocking at your CDN or firewall instead.
Does this affect my Google rankings?
Not directly. AI crawler access is reported as information and never affects your RankForge SEO health score. Google-Extended in particular controls Gemini training only — blocking it does not affect Google Search indexing or ranking at all.
Is it free?
Yes — the check runs with no account. It reads the robots.txt fetched during the same crawl that powers RankForge's full structural audit, which you can run free on up to 100 pages with an account.
Run the ai crawler checker now
Free, no account required. Create a free account afterwards for the full audit and the specific fixes.