The AI Crawler Directory: Main User-Agents, What Each Does, Allow or Block (2026)
Which AI bots should you block? A reference table of the main AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Meta, Amazon, Mistral and more) with each bot's job, robots.txt behaviour, IP-range file and a recommendation.
#ai-crawlers #user-agents #robots-txt