Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

Should You Block or Allow AI Crawlers? A Decision Rule and the robots.txt to Ship

Decide AI crawler access by bot type: training is your call, search bots cite you, user-fetch bots are visitors. A rule by business type, the robots.txt template, Cloudflare settings and a curl check.

By Andrej Ruckij · · 11 min read

Should You Block or Allow AI Crawlers? A Decision Rule and the robots.txt to Ship

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: Decide by bot type, not by “AI”. Training bots turn your content into model weights and send nothing back, so blocking them is a choice you can make either way. Search bots are how AI answers cite you, so allow them. User-fetch bots open a page because a person asked, so don’t block them. The one real judgment call is training, and it comes down to whether your content is the thing you sell or the thing that sells something else. Then check that your CDN or firewall agrees with your robots.txt, because that’s where most accidental blocks happen.

Decide by category

“AI crawler” covers three different jobs (full taxonomy: glossary/ai-crawler):

CategoryWhat it doesWhat you get backDefault
TrainingBuilds model weights from your contentNothing directYour call; block to opt out
Search / retrievalIndexes pages so an AI answer can cite themCitations and referral trafficAllow
User-fetchOpens a page a real person asked aboutA visitorDon’t block

Two of the three answers are fixed. The decision that’s left is training.

The real question: is your content your product?

  • Your content is the product (news, paywalled research, reference data, courses). Training lets a model answer your readers with your work. Lean block, and look at a paid route. This is the publisher logic, and it’s sound for them.
  • Your content is your marketing (SaaS, ecommerce, services, B2B). The site exists to bring in buyers, so being visible in AI answers is an asset and training is a minor call either way.

How many sites over-block is unclear. Elmo’s September 2026 census of the top 5,000 domains found 368 of the 2,771 with a robots.txt (13.3%) blocking at least one AI search crawler, which is 7.4% of all 5,000. So deliberate over-blocking is a minority habit, not the norm. My read: if your site is in that minority and you sell a product or service, you’re probably paying a cost for nothing.

How to decide, by business type

Your businessTraining botsAI search botsUser-fetch botsWhy
News, research, reference or course content with a large archiveblockyour call; the biggest publishers block, but Zhao and Berman found about 7% less traffic in the six weeks after blockingallow unless you’re negotiatingyour content is the product; consider TollBit or a pay-per-crawl waitlist
Small or mid publisher, niche blog with adsblock, or Cloudflare’s Disallow AI Trainingallowallowno licensing buyer is coming; AI search is a source of readers
B2B, SaaS, agency, professional servicesallow or block, minor either wayallowallowbeing named when a buyer asks AI about your category is the point
Online storeallow or blockallowallow, these are shoppersAI shopping answers need your product pages
Local businessallowallowallownothing to protect, everything to gain
Paywalled or member contentblockallow on public pages onlyallow on public pagesthe paywall protects the product; the public pages market it

My read on the training column for marketing sites: blocking training is cheap and doesn’t touch search, so it’s fine if it makes you comfortable. There’s also a real argument for allowing it, which Pixelmojo made when it unblocked training bots in May 2026: a model that learned about you can mention you without searching. Pick one and move on.

This rule doesn’t fit a large publisher already in talks with an AI company. There, blocking everything first is a negotiating move, and the details are in publishers-blocking-ai.

What each choice costs

Allowing training. Your content helps train models that may answer people without sending them to you. AI operators still take far more than they send: Cloudflare Radar’s July 2026 figures, as compiled by TechnologyChecker, show Anthropic crawling about 1,917 pages per referral and OpenAI about 251, against 4.7 for Google. Those ratios fell sharply from July 2025 for Anthropic and OpenAI, but rose for Perplexity, and Cloudflare notes native apps send no referrer, so they overstate the gap (ai-crawler-traffic-impact). For a publisher this is a leak in the core business. For a marketing site it’s mostly noise.

Blocking search bots. You lose the documented route into AI answers. OpenAI’s documentation says sites opted out of OAI-SearchBot won’t be shown in ChatGPT search answers, though they can still appear as navigational links. In practice it’s leakier than that: BuzzStream found 82.4% of news sites blocking OAI-SearchBot were still cited by ChatGPT, and nobody has confirmed why. So a block may cost less visibility than the docs imply, and it also protects less. For a marketing site the trade is still a bad one. More in does-blocking-ai-bots-hurt-seo.

How sites block by accident

When a site blocks search bots by accident, it’s usually one of these:

  • A blanket rule. User-agent: * with Disallow: /, or a “block all AI” plugin, catches OAI-SearchBot along with GPTBot. It also stops Claude-User, which honours robots.txt, so Claude silently can’t open your page when a user asks for it.
  • Cloudflare defaults. Since July 1, 2025, new domains are asked at sign-up whether to block AI crawlers, and existing customers already had a one-click block (since September 2024). Since September 15, 2026, new ad-supported domains get Training set to “Disallow AI Training” and Agent set to “Block on pages with ads”; search stays allowed, and sites without ads default to allowing all three. Existing domains keep their earlier choice, translated into the new controls. Details: does-cloudflare-block-ai-crawlers.
  • A Cloudflare “Block” that reaches Googlebot. Cloudflare’s plain Block settings now also apply to mixed-use crawlers such as Googlebot, Applebot and Bingbot, so a blunt training block can cost you classic search. “Disallow AI Training” publishes a no-training preference in robots.txt instead and keeps those three in search.
  • A firewall that overrides robots.txt. Your Allow loses to a managed “block AI” rule at the edge (robots-txt-vs-waf-ai-bots). Anubis, a popular self-hosted bot wall, blocks all AI-associated user agents by default, search and user-fetch bots included; to follow this page you’d switch it to its moderate policy.

How to set it up

1. Check what you have now. Open https://<your-domain>/robots.txt. Look for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, CCBot, Google-Extended, and any User-agent: * group with Disallow: /. A wildcard block hits every bot that follows robots.txt, search bots included.

2. Write the split. For a marketing site that wants to opt out of training and stay visible in AI answers:

# Training: opt out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Google-Extended
Disallow: /

# Shared training opt-out (Bing from Feb 2027; Google aims for 2027)
User-agent: __usage_ai-training
Disallow: /

# AI search and user fetch: allow
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

Three things to know. Disallowing Google-Extended doesn’t affect Google Search, AI Overviews or AI Mode (they run on Googlebot), but it does opt you out of grounding in Gemini Apps and the Vertex AI Gemini API as well as Gemini training; drop that group if citations in the Gemini app matter to you. Anthropic says the rules have to be in the robots.txt of each subdomain, and Amazon says the same. If you’d rather allow training, delete the first two groups. Every bot name is in ai-crawler-user-agents-directory, and the OpenAI split is explained in openai-crawlers.

3. If you’re on Cloudflare, set the three controls. In the dashboard, open your domain, then Security Settings, and find the AI crawler controls. Set Search to Allow, Agent to Allow, and Training to Disallow AI Training if you’re opting out. Avoid Block on Training unless you’re willing to lose Googlebot and Bingbot crawling as well.

4. Check that the server agrees with the file. robots.txt can say “allow” while your firewall returns 403. From a terminal, compare a normal request with one sent as a bot:

curl -s -o /dev/null -w "%{http_code}\n" https://<your-domain>/
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://<your-domain>/
curl -s -o /dev/null -w "%{http_code}\n" -A "PerplexityBot" https://<your-domain>/

All should return 200 if you meant to allow them. A 403 means a firewall, CDN or hosting rule blocks them regardless of robots.txt. Real bots come from published IP ranges, so a user-agent test is a first check, not proof; verify-ai-bots covers the full method.

5. Look for the bots a month later. Check your server logs or Cloudflare’s AI crawler report for OAI-SearchBot, Claude-SearchBot and PerplexityBot requests. If they’re missing, something upstream still blocks them. OpenAI says search-related robots.txt changes take about 24 hours to apply.

6. Re-check every quarter. Bot names, IP ranges and CDN defaults change; Cloudflare changed its AI defaults in July 2025 and again in September 2026. Each quarter, compare your rules against the current directory, look at your CDN settings again, and rerun the curl check. A stale block list fails silently.

The regulation backdrop

In the EU, the AI Act makes machine-readable opt-outs such as robots.txt legally meaningful for training, so a deliberate training block works as a rights reservation. Since 2 August 2026 the Commission’s AI Office can fine general-purpose AI providers that ignore it. The UK dropped its opt-out proposal in March 2026. Full picture: ai-crawler-regulation-eu-uk. If your content has licensing value, blocking also becomes leverage, but that only works at publisher scale.

Common questions

  • Should I just allow everything? Reasonable for a marketing site that doesn’t mind training. You give up the training opt-out and nothing on visibility.
  • Will blocking training hurt my Google SEO? Not in robots.txt; training bots are separate from Googlebot. On Cloudflare, a plain Block can reach Googlebot, so use Disallow AI Training there. See does-blocking-ai-bots-hurt-seo.
  • Does blocking GPTBot remove me from ChatGPT? No. ChatGPT search uses OAI-SearchBot, which you allow separately (openai-crawlers).

Key takeaways

  • Decide by category: training is your call, allow search, don’t block user-fetch.
  • The training call turns on whether your content is your product or your marketing.
  • Blocking a search bot loses the documented route into AI answers, though citations leak through more than the docs say.
  • Most over-blocking is accidental: wildcard rules, CDN defaults, plain Cloudflare blocks, firewall rules. Reconcile the layers and test with curl.
  • Re-check tokens and CDN defaults every quarter.

Sources

#ai-crawlers#geo#ai-visibility#strategy#robots-txt