#bot-management
13 posts tagged with bot-management.
AI Crawler Access Control: The Complete Guide (2026)
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
The AI Crawler Directory: Main User-Agents, What Each Does, Allow or Block (2026)
Which AI bots should you block? A reference table of the main AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Meta, Amazon, Mistral and more) with each bot's job, robots.txt behaviour, IP-range file and a recommendation.
Anthropic's Crawlers: ClaudeBot, Claude-SearchBot, Claude-User
Anthropic runs ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user-fetch). All three honour robots.txt, including Crawl-delay, which is why a blanket Disallow also stops Claude opening your page for a user.
Blocking AI Crawlers by IP and ASN (for Stealth Scrapers)
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Meta and Amazon AI Crawlers: Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot
Meta runs Meta-ExternalAgent (training or indexing), Meta-WebIndexer (search), Meta-ExternalFetcher (user-fetch) and FacebookExternalHit (link previews); Amazon splits Amazonbot (may train), Amzn-SearchBot and Amzn-User. How to handle each.
How to Block AI Scrapers: The Complete Enforcement Guide (2026)
robots.txt won't stop scrapers that ignore it. The enforcement layer, with working config: rate limiting, bot verification, WAF rules, IP and ASN blocking, and tarpits as a last resort.
OpenAI's Crawlers: GPTBot, OAI-SearchBot, ChatGPT-User (and OAI-AdsBot)
OpenAI runs separate bots for separate jobs: GPTBot (training), OAI-SearchBot (search), ChatGPT-User (user-fetch) and OAI-AdsBot (ads). What each does, whether to block it, and how to verify it by IP range.
Perplexity's Crawlers: PerplexityBot, Perplexity-User, and the Stealth-Crawling Controversy
PerplexityBot indexes for citations; Perplexity-User fetches pages users ask about and generally ignores robots.txt. Plus the August 2025 Cloudflare report that Perplexity crawled sites that blocked it.
How to Verify a Real AI Bot (IP Ranges, Reverse DNS)
User-agent strings are spoofable, so verify AI bots by their published IP-range files, reverse DNS or request signatures, not by name. How to confirm a request really is GPTBot, ClaudeBot or PerplexityBot.
Can robots.txt Stop AI Scrapers?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Do AI Crawlers Respect robots.txt?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Why robots.txt Won't Block AI Bots (and What Actually Does)
Do AI crawlers respect robots.txt? The compliant ones do, user-triggered fetchers often don't, and scrapers ignore it. robots.txt asks; a WAF enforces. Here's the split, bot by bot.
Which AI Bots Should You Block? (And Why robots.txt Won't Stop Them)
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.