Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

Why robots.txt Won't Block AI Bots (and What Actually Does)

Do AI crawlers respect robots.txt? The compliant ones do, user-triggered fetchers often don't, and scrapers ignore it. robots.txt asks; a WAF enforces. Here's the split, bot by bot.

By Andrej Ruckij · · 9 min read

Why robots.txt Won’t Block AI Bots (and What Actually Does)

By Andrej Ruckij · June 16, 2026 · updated November 4, 2026

TL;DR: robots.txt doesn’t block anything. It asks bots to stay away, and only the ones that choose to comply listen. The named crawlers from OpenAI, Anthropic, Google, Perplexity and Amazon do comply; most user-triggered fetchers (Perplexity-User, Google-Agent, often ChatGPT-User) don’t, and scrapers that fake their identity ignore it entirely. A web application firewall (WAF) enforces: it returns a 403 at the edge and never reads robots.txt, so when the two disagree, the firewall wins.

Do AI crawlers respect robots.txt?

Only if their operator chooses to. The spec that defines robots.txt (IETF RFC 9309) says the rules are not a form of access control, and compliance is up to the crawler. In practice the answer splits by bot type:

BotOperatorWhat it doesFollows robots.txt?
GPTBot, OAI-SearchBotOpenAITraining, ChatGPT searchYes
ClaudeBot, Claude-SearchBotAnthropicTraining, Claude searchYes
Googlebot, Google-Extended tokenGoogleSearch; training and Gemini grounding opt-outYes
PerplexityBotPerplexitySearch indexYes
Amazonbot, Amzn-SearchBotAmazonCrawling, searchYes (from a cached copy, see below)
ChatGPT-UserOpenAIOpens a page a user asked forOpenAI says robots.txt rules “may not apply”
Perplexity-UserPerplexityOpens a page a user asked forGenerally ignores it
Google-AgentGoogleActs on a page for a userGenerally ignores it
Amzn-UserAmazonFetches live for a userMay not follow all directives
Meta-ExternalFetcherMetaFetches links a user sharedMay bypass robots.txt
Claude-UserAnthropicOpens a page a user asked forYes, the exception

The logic behind the user-triggered rows: when a person asks an assistant to open a page, most vendors treat the fetch as that person’s visit, not as crawling. Anthropic is the outlier, and that has a side effect. Because Claude-User honours robots.txt, a blanket User-agent: * / Disallow: / quietly stops Claude from opening your page for someone who asked for it.

Changes also aren’t instant. OpenAI says search-related robots.txt updates take about 24 hours to reach its systems, and Amazon’s crawlers work from a copy of your robots.txt cached within the last 30 days. Both Amazon and Anthropic read robots.txt per subdomain, so a file on www doesn’t cover shop.

robots.txt is a request, not a barrier

robots.txt governs one population: the bots that choose to obey it. A Disallow for GPTBot really does stop GPTBot, because OpenAI’s crawler reads the file and complies. It does nothing to a scraper that spoofs its user-agent, rotates IP addresses and never requests the file.

If your goal is “fewer of my words in training sets, from companies that play fair,” robots.txt is the right tool. If your goal is “keep this content out, full stop,” it can’t deliver that alone.

A WAF enforces, and it wins

A web application firewall sits in front of your site and inspects each request before it reaches your server. A request that matches a block rule gets a 403 Forbidden at the edge. The firewall decides on its own and never consults robots.txt.

That leads to the fact most people miss: if a managed WAF rule blocks “all AI crawlers” while your robots.txt says Allow for OAI-SearchBot, the WAF wins. The two aren’t evaluated in some order; they aren’t in the same system. robots.txt lives on your site and depends on the bot’s cooperation. The firewall lives at the edge and depends on nothing. You end up out of ChatGPT search even though your robots.txt invited the crawler in.

So reconcile your layers. If you or your CDN have a “block AI bots” rule switched on, check it against what your robots.txt intends before assuming your search crawlers get through.

What the layers look like

A minimal robots.txt that refuses OpenAI’s training crawler and keeps its search crawler:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

For the full file covering every major vendor, including the Google-Extended caveat, use the template in block-or-allow-ai-crawlers.

That file handles compliant bots. It does nothing about a scraper rotating user-agents from a residential IP pool. For that you need a firewall rule, rate limiting and IP or ASN rules, which how-to-block-ai-scrapers walks through. robots.txt states your intent to the polite; the firewall handles everyone else.

When a polite directive isn’t honoured

In August 2025, Cloudflare reported catching Perplexity crawling sites that had disallowed it, rotating user-agents and networks and using a generic “Chrome on macOS” identity to get past blocks on its declared crawler. Cloudflare removed Perplexity from its verified-bots list and added detection for the behaviour; Perplexity disputed the findings.

Whatever the merits of that dispute, the lesson is structural: a directive only works on whoever chooses to follow it. Once a crawler decides not to, the only response that works is enforcement at the edge.

User-agent strings are not identity

A user-agent is a header any client can set, so a firewall rule keyed on the name catches honest bots and misses the ones pretending. Confirm identity against the operator’s published IP ranges, reverse DNS or a request signature, as described in verify-ai-bots.

Operators that publish IP-range files include OpenAI, Anthropic, Perplexity, Google, Amazon, Apple and Common Crawl. Meta publishes none on its crawler page.

Where access control is heading

The control point keeps moving from your site to your CDN. On July 1, 2025, Cloudflare made AI-crawler blocking the default for new domains at sign-up (a one-click block for existing customers had been there since September 2024). It also launched pay-per-crawl, which uses the HTTP 402 Payment Required status to charge bots for access; Cloudflare’s docs still describe it as currently in closed beta.

Since September 15, 2026, new ad-supported domains on Cloudflare refuse AI training by default (training-only bots such as GPTBot and ClaudeBot are blocked; Googlebot, Bingbot and Applebot stay in search) and block agent bots on pages with ads; search bots stay allowed. New domains without ads get Allow for training and agents. Existing domains keep their earlier choice, translated into the new controls (an old “Block AI” setting becomes Disallow AI Training plus Block on pages with ads). One trap: Cloudflare’s plain “Block” settings now also reach mixed-use crawlers such as Googlebot, Applebot and Bingbot, so a blunt block can cost you classic search. “Disallow AI Training” is the option built to avoid that. Detail: does-cloudflare-block-ai-crawlers.

And the file people reach for next, llms.txt, is not access control either. It’s a content map, and no AI search crawler says it reads yours (does-llms-txt-work).

Key takeaways

  • robots.txt asks and a WAF enforces. The file only governs bots that choose to comply.
  • Named crawlers from OpenAI, Anthropic, Google, Perplexity and Amazon comply. User-triggered fetchers mostly don’t; Claude-User is the exception.
  • Changes take time: about 24 hours for OpenAI search, up to 30 days of caching for Amazon.
  • A firewall block overrides any Allow in robots.txt. Reconcile the two, and on Cloudflare check that a block isn’t reaching Googlebot or Bingbot.
  • Verify bots by IP range, reverse DNS or signature, not by name.

Sources

#ai-crawlers#robots-txt#waf#bot-management#geo#ai-visibility#enforcement