Do AI Crawlers Respect robots.txt?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
By Andrej Ruckij · · 1 min read
Do AI Crawlers Respect robots.txt?
This article has been replaced. The bot-by-bot answer, which crawlers follow robots.txt and which user-triggered fetchers don’t, is now a section of our robots.txt page, next to what actually enforces a block.
Read the current version: Why robots.txt won’t block AI bots (and what actually does)