Can robots.txt Stop AI Scrapers?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
By Andrej Ruckij · · 1 min read
Can robots.txt Stop AI Scrapers?
This article has been replaced. This short answer page repeated our longer guide. The answer hasn’t changed: robots.txt asks, and only a firewall at the edge enforces. The longer guide covers how to verify a bot, what Cloudflare does by default, and the Perplexity case.
Read the current version: Why robots.txt won’t block AI bots (and what actually does)