Which AI Bots Should You Block? (And Why robots.txt Won't Stop Them)
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
By Andrej Ruckij · · 1 min read
Which AI Bots Should You Block? (And Why robots.txt Won’t Stop Them)
This article has been replaced. This page put three things together that each have their own, more current page now. The list of bots, with what each one does and whether to allow it, is in the directory linked below. The decision of what to block is in Should you block or allow AI crawlers?, and why robots.txt can’t enforce any of it is in Why robots.txt won’t block AI bots.
Read the current version: The AI crawler directory: main user-agents, what each does, allow or block