Skip to content
Primores AI wiki

AI Crawler Access Control — Bot Taxonomy, robots.txt vs WAF

AI Crawler Access Control

TL;DR: Not all AI bots are the same. Training bots turn your content into model weights and give nothing back; retrieval/search bots turn it into cited answers with referral links; user-fetch bots fire when a real person asks an AI to open your page (live visitor intent — never block these). The standard play: Disallow training agents in robots.txt, Allow retrieval + user-fetch. But the load-bearing caveat is enforcement: robots.txt only asks compliant bots to stay away; a WAF actually blocks the request at the edge. Real enforcement is the firewall, not the txt file — non-compliant scrapers (spoofed UAs, rotating IPs) ignore robots.txt entirely.

Why this matters

AI bots are now a meaningful share of web traffic, and the access decision has real consequences in both directions. Block too much and you lose AI visibility (your content can’t be cited if the citation bot can’t read it). Block too little and your content trains competitors’ models with no credit or traffic in return. Getting this right requires knowing which kind of bot is knocking — and knowing that the tool most people reach for (robots.txt) is the weakest form of control.

This page sits adjacent to the AI Visibility Audit skill and is reusable for lead-gen audits and GEO projects.

The bot taxonomy (3 core categories + 2 emerging)

The major vendors now expose separate user-agent tokens for each functional role — the taxonomy isn’t just an outside framing; the bots were built around it.

CategoryWhat it doesTraffic/credit back?Default stance
TrainingFetches content to build/refine model weightsNoneBlock (if you don’t want to train models for free)
Retrieval / searchFetches to build a citation index for AI-search answersYes — citations + referral linksAllow
User-fetchFires when a real person asks the AI to open a specific pageYes — represents a live visitorNever block

Two emerging categories a 2026 page should flag (tokens not yet fully stabilized):

  • Agent / action bots — autonomous agents that do tasks (browse, shop, fill forms) on a user’s behalf. Distinct from user-fetch because they take multi-step actions. As of Oct 2026 the first dedicated tokens exist — Google added Google-Agent (Mar 2026) — but most agents identify themselves by cryptographic request signatures (Web Bot Auth / RFC 9421) rather than a robots.txt token: ChatGPT agent signs every request with Signature-Agent: "https://chatgpt.com", and Google is experimenting with Web Bot Auth for Google-Agent.
  • Ads / safety-validation bots — e.g. OpenAI’s OAI-AdsBot, which validates ad landing pages. Neither training, search, nor user-fetch.

Cloudflare’s own classification aligns with this split, though it doesn’t use identical labels: an “AI Crawler” (training) category and an “AI Assistant” (user-driven) category. Note that as of July 1, 2026 Cloudflare merged “AI Search” into a unified “Search” category, and it now tags bots as Direct (one operator) vs Intermediary (a service acting for many end users — the signed-agent case), with Web Bot Auth or IP validation required for verified status.

Enforcement: a WAF acts, robots.txt only asks

This is the most under-known and load-bearing part of access control.

  • robots.txt is a voluntary honor system. Per IETF RFC 9309, compliance is voluntary and “does not constitute access control.” Cloudflare’s analogy: it’s “a Code of Conduct sign posted at a community pool.” A polite bot reads it and self-restricts; nothing forces it to.
  • A WAF (web application firewall) enforces. A request matching a blocked rule gets a 403 (or is dropped) at the edge, before it reaches your origin — regardless of what robots.txt says. The firewall never consults robots.txt.
  • They live at different layers. It’s not that “the WAF rule is parsed before the robots.txt file” — they’re separate mechanisms. If both exist, the WAF wins by construction: it intercepts the request before the bot’s own robots.txt logic is even relevant.
  • robots.txt only governs compliant bots. Non-compliant scrapers — Bytespider is the canonical example, plus stealth crawlers that spoof user-agents, rotate ASNs/IPs, and publish no IP ranges — require WAF rules + IP/ASN blocks + behavioral heuristics.

Bottom line: if you actually need to keep a bot out, the firewall is the enforcement layer; robots.txt is a politeness request that well-behaved bots honor. And UA-string matching alone is weak (UAs are trivially spoofed) — pair it with published IP-range verification (the vendor *.json files below) for anything you want genuinely enforced.

The Perplexity stealth-crawling case (Aug 2025)

A concrete illustration: in August 2025, Cloudflare reported that Perplexity was observed rotating user-agents and source ASNs to disguise crawling — using a generic “Chrome on macOS” UA outside its declared IP ranges when its declared crawler was blocked, and ignoring robots.txt. Cloudflare de-listed Perplexity from its verified-bots list and added managed heuristics. Perplexity denied it, calling the report a “sales pitch.” The episode is the textbook demonstration of “robots.txt only governs bots that choose to comply” — the response to a stealth crawler is WAF heuristics + IP/ASN blocking, not a txt directive.

Current user-agent strings (re-verified Oct 2026)

Dated layer — expect drift. UA tokens, versions, and IP-range publication change. Casing and spelling are load-bearing for robots.txt rules — copy exactly. Re-verify against vendor docs before deploying.

Training crawlers (typically Disallow)

TokenVendorNotes
GPTBotOpenAIGPTBot/1.4 (as of Oct 2026). IP ranges published (openai.com/gptbot.json).
ClaudeBotAnthropicClaudeBot/1.0. IP ranges now published (claude.com/crawling/bots.json, rev. Apr 2026) — older “Anthropic publishes no IP ranges” claims are outdated.
CCBotCommon CrawlNonprofit crawler whose dumps feed many models. Respects robots.txt. IP ranges published (index.commoncrawl.org/ccbot.json); reverse DNS *.crawl.commoncrawl.org (IPv4).
Google-ExtendedGoogleOpt-out token, not a crawler — makes no requests, has no UA of its own. Governs whether Googlebot-fetched content trains future Gemini models and is used for grounding in Gemini Apps / the Vertex AI Gemini API. It does not affect Google Search inclusion — so it does not keep you out of AI Overviews or AI Mode, which run on Googlebot.
Applebot-ExtendedAppleOpt-out token, not a crawler — a signal read by the existing Applebot UA controlling whether content trains Apple’s foundation models. Disallowing it does not remove you from Siri/Spotlight/Safari search (plain Applebot). IP ranges: search.developer.apple.com/applebot.json.
AmazonbotAmazonAmazonbot/0.1. May be used to train Amazon AI models. Respects robots.txt; IP ranges published. Its siblings are not training bots — see Amzn-SearchBot and Amzn-User below.
Meta-ExternalAgentMetameta-externalagent/1.1. Training foundation models or indexing content for Meta products (launched Jul 2024) — so blocking it is not purely a training opt-out. FacebookBot is a separate, older crawler.

Retrieval / search crawlers (typically Allow)

TokenVendorNotes
OAI-SearchBotOpenAIOAI-SearchBot/1.4 (as of Oct 2026). IP ranges published (openai.com/searchbot.json).
Claude-SearchBotAnthropicClaude-SearchBot/1.0. IP ranges in shared bots.json.
PerplexityBotPerplexitySearch/citation indexer (“not used for training”). IP ranges published (www.perplexity.ai/perplexitybot.json) — but see the Aug 2025 verified-bot de-listing above; whether Cloudflare has since restored verified status is unverified as of Oct 2026.
Amzn-SearchBotAmazonAmzn-SearchBot/0.1. Search experiences in Amazon products such as Alexa; Amazon states it is not used for generative-AI training. Separate IP list.
Meta-WebIndexerMetameta-webindexer/1.1. Improves Meta AI search quality; respects robots.txt.

User-fetch agents (never block)

TokenVendorNotes
ChatGPT-UserOpenAIChatGPT-User/1.0. IP ranges published (openai.com/chatgpt-user.json). OpenAI’s docs say robots.txt rules “may not apply” to these user-initiated fetches.
Claude-UserAnthropicClaude-User/1.0. IP ranges in shared bots.json. Anthropic states it honors robots.txt — so a blanket Disallow silently stops Claude from opening your page when a user asks.
Google-AgentGoogleAdded Mar 20, 2026. Used by agents on Google infrastructure to navigate and take actions on a user’s request. Like other Google user-triggered fetchers it generally ignores robots.txt. IP ranges in user-triggered-agents.json; Google is experimenting with Web Bot Auth (https://agent.bot.goog).
Amzn-UserAmazonAmzn-User/0.1. Live fetches for user actions, e.g. Alexa answers needing fresh data; may not follow all robots.txt directives.
Perplexity-UserPerplexityIgnores robots.txt by design (user-triggered). IP ranges published.
Meta-ExternalFetcherMetaFetches individual links at a user’s request (incl. agentic features); may bypass robots.txt.

Others worth knowing

  • OAI-AdsBot (OpenAI) — OAI-AdsBot/1.0, ad landing-page validation; respects robots.txt; IP ranges at openai.com/adsbot.json. New in 2026.
  • Meta-ExternalAds (Meta) — crawls for advertising and other business products; respects robots.txt.
  • Google-CloudVertexBot (Google) — a real crawler (unlike Google-Extended) collecting for Cloud Vertex AI Search; controllable via robots.txt.
  • Agentic-commerce bots — still no dedicated “shopping bot” robots.txt token as of Oct 2026. Agentic commerce runs mainly as protocol/API integrations — OpenAI’s Agentic Commerce Protocol (Instant Checkout deprioritized Mar 24, 2026, checkout returned to merchants; ACP extended to discovery) and Google’s Universal Commerce Protocol (UCP-powered checkout in AI Mode and Gemini for select merchants) — plus general agent identities (Google-Agent, signed ChatGPT agent traffic). Identify agent traffic by signature (Web Bot Auth) or published IP ranges, not by a shopping UA.

robots.txt setup logic — and its one real tradeoff

The recommended pattern: Disallow training agents; Allow retrieval/search + user-fetch agents. Rationale: training takes content and returns nothing; retrieval and user-fetch can drive citations and referral traffic.

The tradeoff is genuine and load-bearing: this clean separation only holds where the vendor exposes separate training vs. search tokens. OpenAI and Anthropic do this well — you can block GPTBot while allowing OAI-SearchBot. Where a vendor uses one token for both, or an opt-out token (Google-Extended, Applebot-Extended) that only governs training use rather than blocking a crawler, the calculus differs. A Disallow aimed at an opt-out token doesn’t “block a crawler” — it changes downstream data use. Users routinely conflate the two mechanisms.

One training opt-out for every vendor: __usage_ai-training (announced October 2026)

Google and Microsoft both said on 30 October 2026 that they will honour a single robots.txt token for opting out of AI training: __usage_ai-training (two leading underscores), written as a user-agent group with Disallow. It comes from an IETF AIPREF draft (draft-kelley-aipref-training-preference-signal), so the details can still change. OpenAI has committed to the signal; Anthropic is reported to be taking part.

  • Dates: Google lists support for “early 2027”. Microsoft starts honouring it on 1 February 2027.
  • Training only. Search indexing and Bing rankings are unaffected. Google’s AI Overviews and AI Mode are opted out separately, in Search Console. Bing grounding has its own settings, with temporary controls in Bing Webmaster Tools from 15 December 2026 and a 1 March 2027 deadline for sites still relying on NoArchive/NoCache.
  • Google-Extended: if a site uses the new token and doesn’t name Google-Extended, Google treats Google-Extended as disallowed too.
  • Existing tokens keep working, so there’s nothing to remove. Until the dates above, keep the per-vendor rules in the table.

What robots.txt and llms.txt can’t do

  • llms.txt is advisory, not access control. It’s an emerging proposed standard — a curated markdown map of your site for LLMs — but it carries the same honor-system limitation as robots.txt and is not consistently honored. Useful for helping compliant AI understand your site; useless as an enforcement mechanism. (See seo/agentic-search-optimization for llms.txt’s visibility role.)

  • The infra-layer shift (Cloudflare, Jul 2025 → Sep 2026). Cloudflare made blocking AI crawlers the default for new domains (Jul 2025; a one-click block for existing customers dated from Sep 2024), and introduced a pay-per-crawl model using HTTP 402 Payment Required plus crawler-price / crawler-max-price headers. The 2026 changes went further:

    • New defaults from Sep 15, 2026 (new domains): two presets. Search is allowed in both. Ad-free sites: training and agents allowed. Ad-supported sites: Training = “Disallow AI Training” (publishes a no-training preference, keeps Googlebot/Bingbot/Applebot in search, and blocks every other training crawler, including the training-only bots of Amazon, Anthropic, Meta and OpenAI) and Agent = block on pages with ads. Existing domains keep their earlier choice, mapped onto the new controls (a legacy “Block AI Bots” becomes Disallow AI Training + block on ad pages).
    • Mixed-use crawlers are now in scope. “Block” and “Block on pages with ads” apply to crawlers that combine search and training — Googlebot, Applebot, Bingbot — so a training block on Cloudflare can now affect search too. Cloudflare added a “Disallow AI Training” option for staying searchable while refusing training, and labels those three “accountable” crawlers because they commit to honoring that preference.
    • Pay-per-crawl is being broadened into the Monetization Gateway (announced Jul 1, 2026; early access): charge any caller for any resource — pages, datasets, APIs, MCP tools — via the x402 protocol (402 response with price, stablecoin payment, retried request with proof).
    • Cloudflare’s own Jul 2026 data: 52% of crawler requests were for AI training (June 2026), up from 22% in spring 2025.

    This reframes “access control” from a per-site config problem into an infra-layer / marketplace problem — increasingly the real control point sits at the CDN, not in your repo. If you’re on Cloudflare, check these settings before assuming your robots.txt is what governs AI access.

  • The direction of travel is cryptographic/verified-bot signing and IP-range verification, because UA-string matching is spoofable. This is now live: ChatGPT agent signs requests (RFC 9421 HTTP Message Signatures), Google is trialling Web Bot Auth for Google-Agent, and Cloudflare requires Web Bot Auth or IP validation for verified status.

A publisher choosing who gets in: Reddit (2024–2026)

Reddit is the clearest case of a large publisher deciding which crawlers and readers get access, and it matters to anyone whose AI visibility leans on Reddit threads.

  • July 2024: Google only. Reddit changed its robots.txt to block crawlers that won’t commit to not using its data for AI training. Bing, DuckDuckGo, Mojeek and Qwant then showed no recent Reddit results or incomplete ones. Google kept access under a data deal reported at around $60M; Reddit said the block was “not at all” related to that deal.
  • October 2026: logged-in first. Reddit announced it is moving to a primarily logged-in experience, with tests in select markets outside the US in Q4 2026 and no global timeline. Search referrals fell by more than 1M a week on average in October. Steve Huffman, quoted by Reuters: “the AI mode does not drive traffic, not for us and not to anybody”, and “Google is crawling us as hard as ever.”

What follows for your own site: the crawl-versus-referral trade that robots.txt manages above is the same one Reddit is making at scale. It still lets Google crawl, because crawl access is part of what it sells, while it closes the human side as referrals fall. For a brand, Reddit’s visibility in AI answers is something Reddit controls, not you. See glossary/retrieval-vs-citation for how much AI answers lean on it.

Key Takeaways

  • Three core bot categories: training (block — no credit back), retrieval/search (allow — drives citations), user-fetch (never block — live visitor). Plus emerging agent and ads-validation bots.
  • robots.txt asks; a WAF enforces. robots.txt only governs compliant bots; the firewall blocks at the edge regardless. Real enforcement = WAF + IP/ASN blocks, not the txt file.
  • UA-string blocking alone is weak — pair with vendor-published IP-range JSON verification.
  • Standard pattern: Disallow training UAs, Allow retrieval + user-fetch UAs — but this clean split only holds where the vendor separates training vs. search tokens.
  • Google-Extended and Applebot-Extended are opt-out signals, not crawlers (no UA, no requests).
  • Anthropic now publishes IP ranges (claude.com/crawling/bots.json); the Perplexity stealth-crawling incident was Aug 2025; Perplexity-User ignores robots.txt by design, while Claude-User honors it.
  • Google-Extended doesn’t touch AI Overviews / AI Mode — those run on Googlebot.
  • Agents are getting identities: Google-Agent (Mar 2026) and signed ChatGPT agent requests — verify by signature/IP, not UA.
  • The control point is moving to the infra layer: since Sep 15, 2026 new ad-supported Cloudflare domains refuse AI training by default (training-only bots blocked, Googlebot/Bingbot/Applebot kept for search) and block agent bots on pages with ads, its blocks now reach mixed-use crawlers (Googlebot, Applebot, Bingbot), and pay-per-crawl is broadening into an x402 Monetization Gateway.

Sources

Reddit access (2024–2026)

Tier 1 — official vendor / infra docs

Tier 2 — reputable trade / security press