Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

Does Cloudflare Block AI Crawlers by Default?

For new domains, partly. Since Sep 15, 2026 new ad-supported domains refuse AI training and block agent bots on ad pages; search stays allowed. Existing domains keep their settings. Edge rules override robots.txt.

By Andrej Ruckij · · 6 min read

Does Cloudflare Block AI Crawlers by Default?

By Andrej Ruckij · June 16, 2026 · updated November 4, 2026

TL;DR: Only for new domains, and only partly. Since September 15, 2026, new ad-supported domains on Cloudflare refuse AI training by default (training-only bots such as GPTBot and ClaudeBot are blocked; Googlebot, Bingbot and Applebot stay in search) and block agent bots on pages with ads; search bots stay allowed. New domains without ads allow all three categories. Existing domains keep their earlier choice, translated into the new controls. If your site is behind Cloudflare, check the controls anyway: an edge block overrides your robots.txt, and a plain “Block” now reaches Googlebot too.

The direct answer

Cloudflare first made AI-crawler blocking a default on July 1, 2025, and called itself the first internet infrastructure provider to do so. From then, every new domain was asked at sign-up whether to allow AI crawlers. Existing customers kept their settings; a one-click block for them had existed since September 2024.

On September 15, 2026 the defaults changed again. New domains now get one of two presets:

ControlSite without adsAd-supported site
SearchAllowAllow
TrainingAllowDisallow AI Training
AgentAllowBlock on pages with ads

“Disallow AI Training” publishes a no-training preference in robots.txt, keeps Googlebot, Bingbot and Applebot allowed for search, and blocks every other training crawler, including the training-only crawlers of Amazon, Anthropic, Meta and OpenAI. Bing doesn’t yet receive the no-training preference through it; Cloudflare says Microsoft’s support is targeted for early 2027.

Existing domains carry their settings over. Cloudflare maps an old “Block AI” setting onto the new controls (Search Allow, Training Disallow AI Training, Agent Block on pages with ads), and leaves domains that had it off allowing everything.

Because all of this runs at the edge, it enforces, and it wins over whatever your robots.txt says.

How the controls work now

On September 15 Cloudflare deprecated the old “Block AI Bots” switch in favour of three controls per domain, Search, Training and Agent, and is replacing its managed robots.txt with Bot Preference Sync, which writes the no-training preference into your robots.txt. The options are Allow, Disallow AI Training (Training only), Block on pages with ads, and Block. A plain Block now also reaches mixed-use crawlers that do both search and training, namely Googlebot, Applebot and Bingbot, so a blunt training block can cost you classic search. Disallow AI Training is the setting that refuses training and keeps you searchable.

Two other changes from 2026:

  • Bot categories. Under the taxonomy Cloudflare introduced on July 1, 2026, the old “AI Search” category is merged into a single Search category, and verified bots are tagged Direct (one operator) or Intermediary (a service acting for many users, such as a signed agent). To be listed as verified, a bot has to pass Web Bot Auth signature checks or IP validation.
  • Charging instead of blocking. Pay per crawl, which answers unpaid crawlers with HTTP 402 Payment Required and a price, is still “currently in closed beta”; you request access through a form, or through your account executive on Enterprise. Cloudflare also announced a Monetization Gateway on July 1, 2026 (early access) that charges callers for pages, datasets, APIs or MCP tools over the x402 protocol.

For scale, Cloudflare reported that 52% of crawler requests were for AI training in June 2026, up from 22% in spring 2025.

Why this matters

An edge rule is the most common way a site ends up invisible in AI search without meaning to. A site owner writes a robots.txt that allows OAI-SearchBot, a Cloudflare rule blocks it anyway, and the firewall wins. The opposite mistake is now possible too: a plain Training block that knocks Googlebot or Bingbot off the site.

What to do about it

  1. Open the AI crawler controls for your domain and read what Search, Training and Agent are actually set to. Don’t assume the September 2026 presets apply; they only do on domains created after that date.
  2. Set Search to Allow if you want to be cited in AI answers.
  3. Use Disallow AI Training, not Block, if you want to refuse training and stay in Google and Bing search.
  4. Decide whether pay per crawl is worth requesting. It’s a closed beta, and Cloudflare publishes no payout figures.
  5. Test from outside. A curl request sent as OAI-SearchBot should return 200; the full check and the robots.txt to pair it with are in block-or-allow-ai-crawlers.

Key takeaways

  • Since Sep 15, 2026, new ad-supported domains refuse AI training and block agent bots on ad pages by default; search stays allowed. New domains without ads allow everything.
  • Existing domains keep their settings unless someone changes them.
  • A plain Block now reaches Googlebot, Applebot and Bingbot; Disallow AI Training keeps them in search.
  • A Cloudflare block overrides robots.txt, so check the dashboard, not just the file.
  • Pay per crawl is still a closed beta.

Sources

#cloudflare#ai-crawlers#faq#waf#pay-per-crawl