Distrify

Blocking AI Crawlers: What It Actually Costs You

September 12, 2026·15 min read
A man in his early thirties in a rust colored t-shirt works at a small wooden desk beside a window in a plant filled home office, laptop open and turned away, in soft morning light

Blocking AI crawlers is not one decision, it is three, and the three do not cost the same thing. Refusing the robot that collects training data changes nothing you can measure on your traffic, while refusing the robot that builds an assistant's search index removes you from its answers, which is the part that was sending you visitors.

The confusion comes from the label. Most of the block lists circulating treat every user agent with "AI" in its name as one category. The companies operating those robots do not: they publish separate tokens for separate jobs and they write down, in their own documentation, what happens when you turn each one off. This article sorts them by quoting those documents, then shows what to check on your own site.

TL;DR

  • OpenAI states the settings are independent: a site can "allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training".
  • Opting out of the search robot has a documented price. Sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links".
  • Google is the exception. Google-Extended does not govern AI Overviews. Google states that "robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search", AI features included.
  • User triggered fetchers are outside the deal. OpenAI writes that for ChatGPT-User "robots.txt rules may not apply", and Perplexity that Perplexity-User "generally ignores robots.txt rules".
  • Blocking is rarer than the debate suggests. On 6 June 2025, out of 3,816 of the top 10,000 domains where Cloudflare found a robots.txt file, 546 of them, about 14 percent, carried any rule targeting AI bots at all.

Table of contents

Blocking AI crawlers: the short answer

If you sell something and you want buyers to find you, blocking the search side of these robots works against you, and the operators say so themselves rather than leaving you to guess. Blocking the training side is a defensible choice that costs you no visibility, because training robots do not decide what an assistant shows today.

The practical version is short. Keep the search robots allowed, decide on the training robots according to what your content is worth to you, and stop expecting a robots file to control the third category, the fetches a human user triggers by asking a question.

None of this replaces the work described in our complete guide to generative engine optimization. Access is the condition, not the method. A robot that can read a page it finds uninteresting will not cite it either.

One label, three different jobs

Every operator in this space runs several robots and gives them different names on purpose. Group them by job rather than by company and the picture becomes readable.

Job What the robot does What blocking it costs you
Training Collects pages that may feed a future model Nothing measurable on today's traffic
Search Builds the index the assistant queries when it answers Your pages stop being eligible as sources
User triggered fetch Opens a page because someone asked a question right now Little, and the rule may not be honoured anyway

The middle row is the one that matters for a site that lives on being found. It is the row people block by accident, because a copied block list puts the training robot and the search robot in the same paragraph.

What each operator documents, in their own words

These are not deductions. Each line below comes from the operator's public documentation, read on 12 September 2026.

Operator Token Job Documented effect of blocking it
OpenAI GPTBot Training Indicates content "should not be used in training generative AI foundation models"
OpenAI OAI-SearchBot Search in ChatGPT Opted out sites "will not be shown in ChatGPT search answers, though can still appear as navigational links"
OpenAI ChatGPT-User User triggered "Because these actions are initiated by a user, robots.txt rules may not apply"
Anthropic ClaudeBot Training Signals "that the site's future materials should be excluded from our AI model training datasets"
Anthropic Claude-SearchBot Search quality "Prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results"
Anthropic Claude-User User triggered "May reduce your site's visibility for user-directed web search"
Perplexity PerplexityBot Search and linking Designed "to surface and link websites in search results on Perplexity", and "not used to crawl content for AI foundation models"
Perplexity Perplexity-User User triggered "Since a user requested the fetch, this fetcher generally ignores robots.txt rules"
Google Googlebot Search, AI features included The control "to manage access to how their sites are crawled for Search"
Google Google-Extended Training and grounding elsewhere Limits "AI training and grounding in some of Google's other systems"

OpenAI is the most explicit about the independence of the two settings: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models". That single sentence removes the dilemma most site owners think they are facing.

Anthropic's page, last updated 7 April 2026, describes three robots and states that its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt", and that IP blocking is not a reliable substitute. It also supports the non standard Crawl-delay directive for sites that want to slow crawling rather than stop it.

Google is the exception that trips everyone up

Google does not follow the pattern of the others, and this is where most published advice is wrong. Google-Extended is not the switch for AI Overviews or AI Mode.

Google's documentation states it directly: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search". Google-Extended is presented in the next sentence as the way "to limit AI training and grounding in some of Google's other systems", which is a different thing.

The consequence is uncomfortable for anyone hoping to be in Search but out of AI Overviews. There is no token for that. The controls Google names for limiting what is shown are nosnippet, data-nosnippet, max-snippet and noindex, and using them has a cost stated in the same documentation: to be eligible as a supporting link in AI Overviews or AI Mode, "a page must be indexed and eligible to be shown in Google Search with a snippet". Remove snippet eligibility and you remove yourself from those answers, and from ordinary featured snippets with them.

The reverse claim is worth killing too. There is no file to add to get in. Google writes that you do not need "to create new machine readable files, AI text files, or markup to appear in these features", and that there is "no special schema.org structured data that you need to add". We went through the practical consequences in how to appear in Google AI Overviews.

Want your content published and distributed on every channel without holding the calendar yourself? Join the waitlist

The blanket rule that quietly removes you from AI answers

The failure mode is banal. Someone copies a block list from a forum thread, pastes twenty user agents into the robots file, and ships it. Six months later the site is absent from assistant answers and nobody connects the two, because nothing broke visibly and traffic from classic search kept working.

Three details make this failure quiet rather than loud.

The first is timing. Changes are not instant, but they are not slow either. OpenAI notes it can take around 24 hours from a robots file update for its systems to adjust, and Perplexity gives the same order of magnitude. The disappearance arrives a day after the change, which is long enough to break the link in your memory.

The second is that there is no report to tell you. An assistant does not send you a message when it stops considering your pages, and the traffic it was sending was probably small enough to look like noise. This is the general measurement problem described in ai visibility tracking and, for competitive questions, in measuring brand share of voice in AI answers.

The third is that the rule you copied was usually written for a publisher whose business is the content itself. Their calculation is not yours.

What blocking protects, and what it does not

Be precise about what the protection actually is, because the gap between belief and effect is where the bad decisions live.

Blocking a training robot signals that your future pages should not enter a training set. That is a real signal and the operators describe it as such. What it does not do is remove what was already collected, stop other companies whose robots you did not list, or prevent a human from copying your text by hand.

Blocking a search robot does something quite different: it takes your pages out of the pool the assistant draws from when it answers a question. In exchange, it protects nothing about training, since those are separate systems on separate tokens.

Blocking a user triggered fetcher mostly protects nothing at all, since the operators state the rule may not apply when the request comes from a person. There is an argument for keeping those tokens allowed anyway: a page fetched because a user asked about you is a page being shown to somebody who is already interested.

Finally, a robots file is a request, not a wall. It works because the operators listed here choose to honour it. Any robot that does not identify itself honestly is not affected by anything you write in that file, which is why sites under real crawl pressure use server side controls instead.

Who has a real reason to block

The case for blocking is not imaginary, and pretending otherwise would be dishonest. It applies when your content is the product rather than the path to the product.

An archive that sells access, a publisher whose revenue comes from readers reaching the page, a research outfit whose reports are licensed: for these, feeding a model that will summarise the work for free is a direct transfer of value. Blocking the training robots is a reasonable defence, and it costs them nothing in search visibility as long as the search robots stay allowed.

There is a second legitimate reason that has nothing to do with principles: crawl load. If a robot is consuming real server resources, rate limiting is the proportionate answer. Anthropic documents support for Crawl-delay for exactly that case.

Most small businesses are in neither situation. If your content exists to bring in customers, the content is the advertisement, not the asset, and being read by the machines that recommend things is the entire point. The mechanism is the one we covered in get mentioned in ChatGPT.

How many sites actually block

The public debate is louder than the practice, and the numbers are worth having in mind before you assume everyone is doing it.

Cloudflare published an analysis on 1 July 2025 based on a 6 June 2025 snapshot of the top 10,000 domains. Of those, a robots file was found on 3,816. Among that group, 546 domains, about 14 percent, carried any allow or disallow directive targeting AI bots. The most blocked token was GPTBot, disallowed by 312 domains, 250 fully and 62 partially, followed by CCBot and Google-Extended. Cloudflare also notes the default that many site owners forget: "bots not listed in a site's robots.txt are effectively allowed by default".

Two things follow. Blocking is a minority behaviour even among large sites, and inaction is itself a choice, since silence means allowed. If you have never opened your robots file, you are already allowing everything, which for most businesses is the right setting, but it deserves to be a decision rather than an accident.

Five minutes to check what your site allows

This check needs no tool and no developer.

  1. Open yourdomain.com/robots.txt in a browser. If you get a 404, nothing is blocked and there is nothing to fix today.
  2. Look for Disallow: / under any user agent. That is the only line that blocks an entire site.
  3. Check whether the search tokens appear: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot. If any of them carries Disallow: /, you have opted out of that assistant's answers.
  4. Check the training tokens separately: GPTBot, ClaudeBot, Google-Extended, CCBot. Blocking these is a content policy choice, not a visibility one.
  5. Check the layer above the file. Google lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" among its best practices. A firewall rule can block a robot your file welcomes, and it will not show up in the file.
  6. If you changed something, wait about a day before drawing conclusions, then follow the page as you would after any publication, as described in what to do after publishing a blog post.

If you run several country versions, do the check on each domain or subdomain separately. Rules do not inherit across hosts, which is the same boundary problem we described in multilingual content for AI search.

What this file cannot do for you

Getting access right moves you from impossible to eligible. It does not move you to cited.

An allowed page still has to be worth quoting: specific, current, attributable, and readable by a machine that will not scroll a carousel to find your specification table. That is a content question, and it is the one we broke down in product content for AI answers.

It also does not change the pace. Google reminds site owners that crawling "can take anywhere from several days to several months". On a young domain, the first weeks are read in impressions rather than clicks, which we described in signs SEO is working before the first clicks. A robots file fixed today does not produce a citation tomorrow.

And it says nothing about whether an assistant is where your buyers actually are. That question is separate, and we looked at the evidence in ChatGPT vs Google search.

FAQ

Does blocking GPTBot remove me from ChatGPT's answers? No. GPTBot is the training robot. The search robot is OAI-SearchBot, and OpenAI states the two settings are independent. Blocking GPTBot while allowing OAI-SearchBot is a documented, supported combination.

If I block Google-Extended, do I disappear from AI Overviews? No. Google states that Googlebot directives are the control for how sites are crawled for Search, and that AI features are part of Search. Google-Extended limits training and grounding in other Google systems.

Can I be in Google Search but out of AI Overviews? There is no token for that. The controls Google names, such as nosnippet and max-snippet, also remove snippet eligibility, and a page must be snippet eligible to appear as a supporting link in AI Overviews or AI Mode.

How long does a change take to apply? OpenAI mentions around 24 hours for search related changes, and Perplexity says up to 24 hours. Google's crawling timeline is separate and can run from several days to several months depending on the page.

Do these robots respect the file at all? The operators quoted here say they do for automated crawling, and Anthropic states its bots honour industry standard robots directives. The stated exception is user triggered fetches, where OpenAI and Perplexity both write that the rules may not apply.

What if my file does not exist? Then nothing is blocked. Cloudflare puts it plainly: robots not listed are effectively allowed by default.

Where this leaves you

Blocking AI crawlers deserves five minutes of attention and one deliberate decision, not a copied list. Separate the three jobs, keep the search robots allowed if you want to be found, treat the training robots as a content policy question, and stop expecting the file to control what a human asks an assistant to fetch.

Then go back to the part that actually decides whether you get cited: publishing often enough, on the subjects your buyers search, in every place they look. The overview is in our content distribution strategy, and the counterpart on AI generated content, which raises the same access questions, is in what Google really says about AI content.

Join the waitlist

Sources