Should I Allow AI Crawlers?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
By Andrej Ruckij · · 1 min read
Should I Allow AI Crawlers?
This article has been replaced. This short answer page repeated our longer guide, which decides by bot type, covers the accidental blocks (Cloudflare defaults, firewall rules that override robots.txt) and gives a robots.txt you can copy.
Read the current version: Should you block or allow AI crawlers?