Anthropic's Crawlers: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic’s Crawlers: ClaudeBot, Claude-SearchBot, Claude-User

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: Anthropic runs three bots: ClaudeBot (training), Claude-SearchBot (search) and Claude-User (fetches a page when a Claude user’s question needs it). All three honour robots.txt, and Anthropic also supports Crawl-delay. That compliance has a side effect: a blanket User-agent: * / Disallow: / silently stops Claude opening your page for a user. Set rules per bot, on every subdomain, and prefer robots.txt over IP blocking, which Anthropic says may not reliably guarantee an opt-out.

Part of the AI crawler directory. Anthropic splits its bots by job like OpenAI does, but a few of its rules work differently.

The three Anthropic bots

User-agent	Job	robots.txt	Recommendation
`ClaudeBot`	Collect web content that may be used for model training	Honours it	Block to opt out of training
`Claude-SearchBot`	Index content to improve search answers	Honours it	Allow
`Claude-User`	Fetch a page when a user’s question calls for it	Honours it	Allow

Anthropic’s page gives the bare names for robots.txt use. It doesn’t print full user-agent strings, so match on the token.

Claude-User honours robots.txt, and that cuts both ways

Most vendors’ user-fetch agents don’t reliably follow robots.txt: OpenAI says rules “may not apply” to ChatGPT-User, and Perplexity-User generally ignores them. Claude-User is different. It honours the file, which means a site-wide Disallow: / under User-agent: * keeps Claude from opening your page even when a person asked it to. Nothing tells you this happened; the user just gets an answer without your page in it.

If you want a broad block on unknown bots, write explicit groups for the Anthropic bots you want to keep, since under the robots.txt standard (RFC 9309) a crawler uses the * group only when no group names it.

Crawl-delay works here

Anthropic supports the non-standard Crawl-delay extension, so you can slow ClaudeBot down instead of blocking it:

User-agent: ClaudeBot
Crawl-delay: 1

Not every vendor does this. Amazon, for one, says its crawlers don’t support Crawl-delay (meta-amazon-ai-crawlers).

Every subdomain needs its own file

Anthropic asks you to add the rules “for every subdomain that you wish to opt out from.” A Disallow in example.com/robots.txt doesn’t cover blog.example.com or shop.example.com.

IP blocking: the caveat, and the unlabelled list

Anthropic publishes its crawler IP addresses at claude.com/crawling/bots.json. If a request’s source IP is on that list, it’s from Anthropic. Two limits:

My read of the file: it’s one list for all three bots, with nothing saying which range belongs to which. You can confirm a request is Anthropic’s, but not which bot sent it, so you can’t use it to block ClaudeBot by IP while letting Claude-User through.
Anthropic says IP-based blocking “may not work correctly or persistently guarantee an opt-out”, and that it can stop the bots from reading your robots.txt. It recommends robots.txt instead.

So the list is for verification, not for writing per-bot firewall rules.

Anthropic’s robots.txt lines

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

The Claude-User line matters more here than the ChatGPT-User line does for OpenAI, because it protects you if you also have a broad * block. For the full cross-vendor file, use the template in block-or-allow-ai-crawlers.

Key takeaways

Three bots: ClaudeBot (training), Claude-SearchBot (search), Claude-User (user-fetch). All honour robots.txt.
Because Claude-User complies, a blanket Disallow: / silently stops Claude opening your page for a user.
Anthropic supports Crawl-delay, so you can throttle ClaudeBot instead of blocking it.
Put robots.txt rules on every subdomain you want to opt out.
bots.json verifies that a request is Anthropic’s but doesn’t say which bot; Anthropic warns IP blocking may not reliably guarantee an opt-out.

ai-crawler-user-agents-directory: the cross-vendor bot table
block-or-allow-ai-crawlers: the decision and the full robots.txt template
openai-crawlers: the equivalent for OpenAI’s bots
glossary/gptbot: the parallel training-crawler concept

Sources

Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?: the three bots and their jobs, robots.txt compliance, Crawl-delay support, per-subdomain rule, IP-blocking caveat, bots.json; checked November 4, 2026
OpenAI: Overview of OpenAI crawlers: ChatGPT-User robots.txt “may not apply”
Perplexity: Perplexity crawlers: Perplexity-User “generally ignores robots.txt rules”
Amazon: Amazonbot and other Amazon crawlers: no Crawl-delay support
RFC 9309: Robots Exclusion Protocol: the * group applies only when no group matches