Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

Does Blocking AI Bots Hurt Your SEO or AI Visibility?

Blocking AI training bots in robots.txt doesn't hurt Google SEO, and Google-Extended doesn't touch AI Overviews. Blocking AI search bots, or a plain Cloudflare block that reaches Googlebot, does cost visibility.

By Andrej Ruckij · · 5 min read

Does Blocking AI Bots Hurt Your SEO or AI Visibility?

By Andrej Ruckij · June 16, 2026 · updated November 4, 2026

TL;DR: Blocking AI training bots (GPTBot, ClaudeBot, CCBot) in robots.txt doesn’t hurt your Google rankings, and disallowing Google-Extended doesn’t remove you from Google Search, AI Overviews or AI Mode. Blocking AI search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) does cost AI visibility: you lose the documented route into those answers, even if some citations still leak through. The trap is Cloudflare, where a plain “Block” now reaches Googlebot and Bingbot too. Block by category, not with one switch.

The direct answer

It depends on which bots you block:

  • Training bots (GPTBot, ClaudeBot, CCBot): no effect on Google SEO. They’re separate from Googlebot, which keeps crawling and ranking you. You only opt out of model training.
  • Search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot): costs AI visibility. OpenAI’s documentation says sites opted out of OAI-SearchBot won’t be shown in ChatGPT search answers, though they can still appear as navigational links.
  • User-fetch bots (ChatGPT-User, Claude-User, Perplexity-User): these requests come from people mid-question, so blocking them turns visitors away. A robots.txt Disallow won’t reliably stop most of them (OpenAI says robots.txt rules “may not apply” to ChatGPT-User, and Perplexity-User generally ignores it). Claude-User does honour it, so a blanket User-agent: * / Disallow: / silently stops Claude opening your page when a user asks.

How much a search-bot block really costs

Less than the documentation suggests, and nobody knows exactly why. BuzzStream analysed about 4 million AI citations and found that 82.4% of news sites blocking OAI-SearchBot were still cited by ChatGPT. Its author guesses at copies via Common Crawl, bots ignoring robots.txt, or answers built from search snippets, and admits the mechanism is unknown. So the fair summary is that you lose the documented route in, not that you vanish. My read: that leak doesn’t make blocking search bots a good idea for a site that wants to be found. It makes the block a weak protection as well as a cost.

What about Google’s own AI?

Google-Extended is narrower than its name. It’s an opt-out token, not a crawler, and it controls whether content Googlebot fetched is used to train Gemini and to ground answers in Gemini Apps and the Vertex AI Gemini API. It doesn’t affect Google Search, AI Overviews or AI Mode, which run on Googlebot. Disallowing it costs no rankings and no AI Overviews presence; it can cost citations inside the Gemini app. No robots.txt token keeps you in Google Search but out of AI Overviews.

The Cloudflare exception

The “no effect on Google SEO” answer assumes you block in robots.txt. On Cloudflare, the plain “Block” settings now also apply to mixed-use crawlers that do both search and training: Googlebot, Applebot and Bingbot. A blunt block there can cost search visibility. Cloudflare’s “Disallow AI Training” publishes a no-training preference in robots.txt instead, keeps those three in search, and blocks the other training crawlers. One gap: Bing doesn’t yet receive that preference through it. See does-cloudflare-block-ai-crawlers.

Why people ask this

The worry is that “blocking AI” is one switch that trades privacy for invisibility. It isn’t. Google rankings, citations in AI answers and model training run on different crawlers. A blanket block hits all three; a targeted one costs nothing on Google and only opts you out of training.

How to apply it

The low-regret setup is to block training and allow search and user-fetch. OpenAI’s part of it looks like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

The full file for every vendor, with the Cloudflare settings and a curl check that your firewall agrees with it, is in block-or-allow-ai-crawlers. This clean split works where a vendor separates training and search bots (OpenAI and Anthropic do). Also check that no CDN or firewall rule overrides your intent (robots-txt-vs-waf-ai-bots).

Key takeaways

  • Blocking AI training bots in robots.txt has no effect on Google SEO.
  • Disallowing Google-Extended doesn’t affect Search, AI Overviews or AI Mode; it opts you out of Gemini training and Gemini Apps / Vertex grounding.
  • Blocking AI search bots loses the documented route into AI answers, though BuzzStream found most blocking news sites still cited.
  • On Cloudflare, a plain Block can reach Googlebot and Bingbot; use Disallow AI Training to stay searchable.
  • Block by category, never with a blanket rule.

Sources

#ai-crawlers#seo#geo#faq#ai-visibility