SEONIB SEONIB

Your robots.txt Is Making an AI Decision Nobody Reviewed

Author: SEONIB Editorial Date: 2026-11-13 13:37:39
Your robots.txt Is Making an AI Decision Nobody Reviewed

There is a file on your site that decides whether you exist inside ChatGPT, Claude, and Perplexity. It is four kilobytes of plain text, it was probably written years ago, and in most companies nobody has looked at it since the launch.

That file is now a strategic decision, and most sites are making it by accident.

Two jobs, one file

AI providers run at least two kinds of crawler, and they do opposite jobs:

  • Training crawlers collect text to build future models.
  • Retrieval crawlers fetch pages live to answer a user’s question, with a citation.

Blocking the first is a licensing and content-control decision. Blocking the second is a visibility decision. The mistake is treating them as the same act.

OpenAI’s own documentation is explicit: to be included in ChatGPT’s summaries and snippets, you must not block OAI-SearchBot. Separately, disallowing GPTBot is how you opt out of training. Two switches, two outcomes.

The bots that actually matter

Bot Operator Job What blocking costs you
GPTBot OpenAI Training Opts your content out of model training; search visibility unaffected
OAI-SearchBot OpenAI Search retrieval Removed from ChatGPT search answers and citations
ChatGPT-User OpenAI Live fetch on user request Removed from real-time browsing answers
ClaudeBot Anthropic Training + retrieval Removed from Claude answers
PerplexityBot Perplexity Retrieval index Removed from Perplexity citations entirely
Google-Extended Google Gemini/AI training No effect on Google Search ranking
Googlebot Google Search index Removed from Google Search, including AI Overviews eligibility

The Google row is the one people get wrong most often. Google-Extended is a training control. It does not affect classic ranking, and it does not remove you from AI Overviews. If you want out of Google’s generative features, the lever is Googlebot plus noindex, which is a much bigger decision.

How common is the accidental block

A survey of robots.txt across 1,069 top websites found roughly one in five blocks at least one major AI crawler. Among the 268 sites blocking at least one training crawler, 87% blocked two or more and 74% blocked three or more.

That pattern is telling. Individual one-off rules would produce scattered results; near-identical block lists across hundreds of sites suggest a policy copied from a template, a CDN preset, or a security plugin — applied without anyone deciding what it meant for retrieval.

The three failure modes

1. Blocking both and calling it “protecting content.” You keep your text out of training and also out of the answers your customers read. The first effect is invisible; the second shows up as missing citations six months later.

2. Relying on a CDN default. Some bot-management presets block “AI crawlers” as a category. That category includes retrieval bots. Audit the preset, don’t trust the label.

3. Blocking to stop server load. Retrieval crawlers can be aggressive. But robots.txt is the wrong instrument for rate control — it is all-or-nothing, and the cost is total invisibility. Use CDN rate limits, caching, and a separate crawl-friendly text path instead.

What about llms.txt?

The idea is a plain-text map at your domain root listing the pages that matter most, described in human language. It costs twenty minutes to write and it is genuinely useful as documentation — for the models that read it, and for your own team.

The honest caveat: it is a proposal, not a standard with enforcement. Do not treat it as a substitute for crawlability, structured content, or sitemaps. It is a hint, and hints only work when the fundamentals already do.

A five-minute audit

  1. Fetch yourdomain.com/robots.txt and read the user-agent blocks.
  2. Confirm OAI-SearchBot, PerplexityBot, ClaudeBot, Googlebot are allowed on the content paths that matter.
  3. Decide explicitly on GPTBot — training opt-out, yes or no — and write down who decided why.
  4. Check whether your CDN, WAF, or bot-management plugin overrides those rules.
  5. Verify with a fetch: request a key article URL using the bot’s user agent and see what comes back. A 403 from the edge is the silent version of a robots block.

Why this is a content problem in disguise

Blocking a retrieval bot does not degrade your page’s quality; it removes the page from consideration. You then spend months improving content that no model is allowed to read.

The fix is cheap and unglamorous, which is exactly why it is worth doing once, properly, and documenting the decision so the next plugin install does not silently reverse it.

Once access is open, the work shifts to making pages quotable — dated facts, tables, clear entity naming. SEONIB builds articles with those fields by default and publishes them across language versions, so the pages your retrieval crawlers can now reach are the same pages an answer engine can actually lift.

Share Article

Related Articles

Recommended Reading

Ready to Get Started?

Experience our product immediately and explore more possibilities.