Does Cloudflare Block AI Crawlers by Default?

Yes. Since July 2025 Cloudflare blocks AI crawlers by default for new sites and offers one-click blocking plus pay-per-crawl. If you're on Cloudflare, check this setting — it can override your robots.txt.

By Andrej Ruckij · · 2 min read

Does Cloudflare Block AI Crawlers by Default?

By Andrej Ruckij · June 16, 2026

TL;DR: Yes. In July 2025 Cloudflare began blocking AI crawlers by default for new sign-ups and offers existing customers one-click AI blocking plus a pay-per-crawl option. If your site is behind Cloudflare, check this setting — a default block can override your robots.txt and make you invisible to AI search.

The direct answer

Cloudflare made AI-crawler blocking a default in July 2025 — the first major infrastructure provider to do so. The relevant facts:

  • New sites going through Cloudflare can have AI crawlers blocked by default.
  • Existing customers get one-click controls (AI Crawl Control) to block or allow categories of AI bots.
  • Pay-per-crawl lets sites charge crawlers for access via HTTP 402 Payment Required instead of simply blocking.

Because this operates at the firewall/edge layer, it enforces — and it overrides robots.txt.

Why this matters

This is the most common cause of accidental AI-invisibility in 2026. A site owner carefully writes a robots.txt that allows OAI-SearchBot, but a Cloudflare default rule blocks “all AI crawlers” — and the firewall wins, so they’re cut out of ChatGPT search without realizing it. The control point has moved to the CDN; your repo’s robots.txt no longer has the final say.

What to do about it

  1. Check your Cloudflare dashboard — look at AI Crawl Control / bot-management settings to see what’s currently blocked.
  2. Reconcile with your intent — if you want AI search visibility, ensure retrieval bots (OAI-SearchBot, PerplexityBot) are allowed at the Cloudflare layer, not just in robots.txt.
  3. Decide on pay-per-crawl — if you’d rather monetize training access than block it outright.
  4. Verify end-to-end — test that your pages are actually reachable by AI search bots (an audit that uses spoofed user-agents catches CDN hard-blocks; see tools/ai-visibility-audit).

Key takeaways

  • Yes — Cloudflare blocks AI crawlers by default (since July 2025) and offers one-click controls + pay-per-crawl.
  • A Cloudflare block overrides robots.txt — the firewall wins.
  • If you want AI visibility, check the Cloudflare setting, not just your robots.txt.

Sources