Anthropic's Crawlers: ClaudeBot, Claude-SearchBot, Claude-User
Anthropic runs ClaudeBot (training), Claude-SearchBot (search), and Claude-User (user-fetch). All respect robots.txt, and — updated in 2026 — Anthropic now publishes IP ranges for all three.
Anthropic’s Crawlers: ClaudeBot, Claude-SearchBot, Claude-User
By Andrej Ruckij · June 17, 2026
TL;DR: Anthropic runs three bots mirroring the standard split — ClaudeBot (training), Claude-SearchBot (search), Claude-User (user-fetch). All respect robots.txt, and — an important 2026 update — Anthropic now publishes IP ranges for all three at
claude.com/crawling/bots.json, so older guidance saying “Anthropic doesn’t publish ranges” is outdated.
A cluster under the AI crawler directory. Anthropic, like OpenAI, separates its bots cleanly by function.
The three Anthropic bots
| User-agent | Job | Respects robots.txt | IP ranges | Recommendation |
|---|---|---|---|---|
ClaudeBot | Train Anthropic’s models | Yes | claude.com/crawling/bots.json | Block to opt out of training |
Claude-SearchBot | Index for Claude’s search/citations | Yes | (shared bots.json) | Allow (drives citations) |
Claude-User | Fetch a page a user asked Claude about | Yes | (shared bots.json) | Never block (it’s a visitor) |
ClaudeBot — the training crawler
ClaudeBot (ClaudeBot/1.0) collects content to train Anthropic’s Claude models. Same logic as OpenAI’s GPTBot: block it if you don’t want to contribute to model training for free. It respects robots.txt, so the block is honored.
Claude-SearchBot — allow it
Claude-SearchBot indexes content so Claude can cite it in answers with a link back. Allow it for the same reason you’d allow OAI-SearchBot — it’s how you stay visible and citable in Claude’s responses. Blocking training while allowing search is the clean, standard policy.
Claude-User — never block
Claude-User fires when a real person asks Claude to open your page. It’s a visitor, not a harvester. Don’t block it.
The 2026 IP-range update
The notable change: Anthropic now publishes a verifiable IP-range file (claude.com/crawling/bots.json, covering all three bots, revised in 2026). A lot of older “how to block Claude” guides claim Anthropic doesn’t publish ranges — that’s no longer true. Use the JSON to verify a bot really is Anthropic’s rather than a scraper using the name (user-agent strings are spoofable).
Recommended robots.txt
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Key takeaways
- Three bots: ClaudeBot (training), Claude-SearchBot (search), Claude-User (user-fetch).
- Block training, allow search, never block user-fetch — same policy as OpenAI.
- Anthropic now publishes IP ranges (
claude.com/crawling/bots.json) — retire the old “no ranges” claim.
Related articles
- ai-crawler-user-agents-directory — the full cross-vendor bot table
- openai-crawlers — the equivalent for OpenAI’s bots
- which-ai-bots-to-block — the overall allow/block policy
- glossary/gptbot — the parallel training-crawler concept