Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

The AI Crawler Directory: Main User-Agents, What Each Does, Allow or Block (2026)

Which AI bots should you block? A reference table of the main AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Meta, Amazon, Mistral and more) with each bot's job, robots.txt behaviour, IP-range file and a recommendation.

By Andrej Ruckij · · 11 min read

The AI Crawler Directory: Main User-Agents, What Each Does, Allow or Block (2026)

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: Sort every AI bot into one of three jobs. Training bots turn your pages into model weights and send nothing back, so block them if you want to opt out. Retrieval or search bots index you so an AI answer can cite and link you, so allow them. User-fetch bots fire when a person asks an AI to open your page, so treat them as visitors and leave them alone. The tables below give each main bot’s exact token, whether it follows robots.txt, and where its vendor publishes IP ranges. Tokens drift, so re-check vendor docs before you deploy.

What you’ll learn

  • The exact user-agent token for the main AI crawlers, by vendor
  • Which job each one does: training, retrieval, user-fetch, or opt-out token
  • Whether each follows robots.txt, and where to verify it by IP range
  • An allow or block recommendation per bot
  • Which “bots” are opt-out tokens that never visit your site

Why this matters now

Training traffic is the biggest part of it. Cloudflare reports that 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025. That’s the traffic that gives you nothing back, and it’s the part robots.txt lets you refuse.

How to read this directory

Three categories drive every decision (full taxonomy: glossary/ai-crawler):

  • Training turns your content into model weights. No citation, no traffic. Block to opt out.
  • Retrieval / search indexes you to cite in AI answers, with a link. Allow.
  • User-fetch fires when a real person asks an AI to open your page. Leave it alone; it’s a visitor.

Two cautions before the tables. User-agent strings are spoofable, so verify anything you care about against the operator’s published IP-range file. And robots.txt only governs bots that choose to comply; for the rest you need a firewall (robots-txt-vs-waf-ai-bots).

Training crawlers (block to opt out of training)

User-agentVendorFollows robots.txtIP rangesNotes
GPTBotOpenAIYesopenai.com/gptbot.jsonGPTBot/1.4 as of November 2026. See openai-crawlers
ClaudeBotAnthropicYesclaude.com/crawling/bots.jsonOne unlabelled list for all three Anthropic bots. Anthropic also honours Crawl-delay. anthropic-crawlers
CCBotCommon CrawlYesindex.commoncrawl.org/ccbot.jsonCCBot/2.0. IPv4 reverse DNS resolves to *.crawl.commoncrawl.org. Its open dataset feeds many models. glossary/ccbot
Meta-ExternalAgentMetaNo bypass statedNot publishedmeta-externalagent/1.1. Meta lists training foundation models and indexing content to improve its products, so blocking it isn’t a pure training opt-out. meta-amazon-ai-crawlers
AmazonbotAmazonYesdeveloper.amazon.com/amazonbot/ip-addresses/Amazonbot/0.1. Content “may be used to train Amazon AI models”. No Crawl-delay support
MistralAI-TrainingMistralCan be disallowedNot on the robots pageMistralAI-Training/1.0. Builds training datasets; not used for search or live answers
Google-ExtendedGoogleToken onlyNone (no requests)Opt-out token, not a crawler. Covers Gemini training and grounding in Gemini Apps / Vertex AI. google-ai-crawlers
Applebot-ExtendedAppleToken onlyNone (no requests)Opt-out token read against what Applebot already crawled; covers training of Apple’s foundation models

Retrieval / search crawlers (allow, they cite you)

User-agentVendorFollows robots.txtIP rangesNotes
OAI-SearchBotOpenAIYesopenai.com/searchbot.jsonOAI-SearchBot/1.4. Allow to appear in ChatGPT search answers
Claude-SearchBotAnthropicYesclaude.com/crawling/bots.jsonAllow for Claude search citations
PerplexityBotPerplexityYesperplexity.ai/perplexitybot.jsonPerplexityBot/1.0. Not used for training. See the August 2025 caveat in perplexity-crawlers
Amzn-SearchBotAmazonYesdeveloper.amazon.com/amazonbot/searchbot-ip-addresses/Amzn-SearchBot/0.1. Search in Amazon products such as Alexa; not used for generative-AI training. If you don’t name it, it follows the rules you give other search bots
Meta-WebIndexerMetaNo bypass statedNot publishedmeta-webindexer/1.1. Improves Meta AI search; Meta says allowing it helps Meta AI cite and link you
MistralAI-IndexMistralNot statedNot on the robots pageMistralAI-Index/1.0. Indexes for Mistral search in Vibe; not used for training
DuckAssistBotDuckDuckGoYesduckduckgo.com/duckassistbot.jsonDuckAssistBot/1.2. Fetches pages in real time for DuckDuckGo’s AI-assisted answers, which cite sources; not used for training. A Disallow takes effect after 72 hours and doesn’t affect organic rankings

User-fetch agents (leave alone, they’re visitors)

User-agentVendorFollows robots.txtIP rangesNotes
ChatGPT-UserOpenAINot guaranteedopenai.com/chatgpt-user.jsonChatGPT-User/1.0. OpenAI says robots.txt rules “may not apply” to these user-initiated fetches
Claude-UserAnthropicYesclaude.com/crawling/bots.jsonBecause it honours robots.txt, a blanket User-agent: * / Disallow: / stops Claude opening your page when a user asks
Perplexity-UserPerplexityGenerally noperplexity.ai/perplexity-user.jsonPerplexity-User/1.0. Perplexity says it “generally ignores robots.txt rules”
Google-AgentGoogleGenerally nouser-triggered-agents.jsonAgents on Google infrastructure that navigate and act for a user. Google is experimenting with Web Bot Auth (https://agent.bot.goog)
Amzn-UserAmazonNot fullydeveloper.amazon.com/amazonbot/live-ip-addresses/Amzn-User/0.1. Live fetches for user requests such as Alexa answers; “may not follow all robots.txt directives”
Meta-ExternalFetcherMetaNot guaranteedNot publishedmeta-externalfetcher/1.1. Fetches links at a user’s request, including agentic features; Meta says it “may bypass robots.txt rules”
MistralAI-UserMistralNot detailedSeparate JSON fileMistralAI-User/1.0. Visits a page when a Vibe user asks a question and links the source; not used for crawling or training

The “bots” that aren’t crawlers

Three entries make no requests at all:

  • Google-Extended is a robots.txt opt-out token. It tells Google not to use Googlebot-fetched content for training Gemini models or for grounding answers in Gemini Apps and the Vertex AI Gemini API. It doesn’t block any crawling and doesn’t affect Google Search, including AI Overviews and AI Mode, which run on Googlebot. (google-ai-crawlers)
  • Applebot-Extended does the same job for Apple. Apple says it “does not crawl webpages”; pages that disallow it can still appear in search results.
  • __usage_ai-training is the AIPREF shared training-preference signal, written as a user-agent line. Google says it intends to support it in early 2027. Cloudflare’s “Disallow AI Training” setting already publishes a no-training preference in robots.txt.

Disallowing these changes how your content is used, not who can fetch it. Mixing up opt-out tokens and crawlers is an easy mistake to make.

Other bots worth knowing

  • OAI-AdsBot (OpenAI, OAI-AdsBot/1.0) visits only pages submitted as ads, and OpenAI says the data isn’t used to train foundation models. IP ranges at openai.com/adsbot.json. OpenAI’s page says nothing about robots.txt for it.
  • Google-CloudVertexBot (Google) handles crawls that site owners request to build Vertex AI Agents. It doesn’t cite you anywhere and has no effect on Google Search. It crawls from the ranges in common-crawlers.json. Most sites never need a rule for it.
  • FacebookExternalHit (Meta) builds link previews when your page is shared. Meta says it may bypass robots.txt for security or integrity checks.
  • Meta-ExternalAds (Meta, meta-externalads/1.1) crawls for advertising and other business products; you can disallow it in robots.txt.
  • Bytespider (ByteDance) is the usual example of a scraper reported to ignore robots.txt. Only a firewall rule stops it. (glossary/bytespider)
  • Shopping and agent traffic. My read: there’s still no dedicated shopping-bot token. ChatGPT agent signs its requests with HTTP Message Signatures (RFC 9421), and Google-Agent is trialling Web Bot Auth, so identify this traffic by signature or IP range rather than by user-agent. See how-do-ai-shopping-agents-work.

Vendor quirks that change how you write rules

  • Anthropic supports Crawl-delay, needs a robots.txt on each subdomain you want to opt out, and warns that IP blocking may not reliably guarantee an opt-out.
  • Amazon doesn’t support Crawl-delay, uses a robots.txt copy cached within the last 30 days, and reads each subdomain’s file separately.
  • OpenAI says search-related robots.txt changes take about 24 hours to apply. When GPTBot and OAI-SearchBot are both allowed, it may use one crawl for both.
  • Common Crawl publishes IP ranges and reverse DNS for IPv4, so you can verify CCBot like the big vendors’ bots.

Turning this into a robots.txt

The full template, with the Google-Extended caveat, lives in one place: block-or-allow-ai-crawlers. The vendor pages linked above show only their own lines.

Two notes before you copy it. You don’t need to list user-fetch agents at all, and a Disallow wouldn’t reliably stop most of them: ChatGPT-User, Perplexity-User, Google-Agent, Amzn-User and Meta-ExternalFetcher may not follow robots.txt, while Claude-User does. And a blanket User-agent: * / Disallow: / takes out the search and user-fetch bots along with the training ones.

Common questions

  • Will blocking GPTBot hurt my ChatGPT visibility? No. ChatGPT search uses OAI-SearchBot, a separate bot you can allow. See openai-crawlers.
  • Does blocking Google-Extended hurt my Google ranking? No. It’s an opt-out token, not Googlebot. See google-ai-crawlers.
  • Is Cloudflare already blocking these for me? Possibly. Since September 15, 2026, new ad-supported domains on Cloudflare refuse AI training by default (training-only bots such as GPTBot and ClaudeBot are blocked; Googlebot, Bingbot and Applebot stay in search) and block agent bots on pages with ads; search bots stay allowed. Existing domains keep their earlier choice, translated into the new controls (an old “Block AI” setting becomes Disallow AI Training plus Block on pages with ads). See does-cloudflare-block-ai-crawlers.
  • Does blocking hurt SEO? Blocking training bots doesn’t touch classic search. Blocking search bots costs you the documented route into AI answers. See does-blocking-ai-bots-hurt-seo.

Maintenance note

Tokens, version numbers and IP-range files change often. This directory was last checked against each vendor’s own page on November 4, 2026. Re-verify before deploying and refresh quarterly; a stale block list fails quietly.

Key takeaways

  • Classify every bot as training (block to opt out), retrieval (allow) or user-fetch (leave alone).
  • Verify by published IP range, not the user-agent string. Common Crawl, OpenAI, Anthropic, Perplexity, Amazon, Apple and DuckDuckGo publish ranges; Meta doesn’t on its crawler page.
  • Google-Extended, Applebot-Extended and the AIPREF __usage_ai-training line are signals, not crawlers.
  • robots.txt governs compliant bots only; scrapers like Bytespider need a firewall.
  • Keep one robots.txt template and re-check the tokens quarterly.

Sources

#ai-crawlers#user-agents#robots-txt#bot-management#reference