#waf
11 posts tagged with waf.
AI Crawler Access Checklist: 8 Steps for Site Owners
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
AI Crawler Access Control: The Complete Guide (2026)
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
AI Crawler Tarpits and Honeypots: Nepenthes, Anubis, and Cloudflare AI Labyrinth
When blocking isn't enough, some sites trap or slow AI crawlers instead. What Nepenthes, Anubis and Cloudflare AI Labyrinth actually do, and the search-visibility costs of each.
Blocking AI Crawlers by IP and ASN (for Stealth Scrapers)
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
How to Block AI Scrapers: The Complete Enforcement Guide (2026)
robots.txt won't stop scrapers that ignore it. The enforcement layer, with working config: rate limiting, bot verification, WAF rules, IP and ASN blocking, and tarpits as a last resort.
Perplexity's Crawlers: PerplexityBot, Perplexity-User, and the Stealth-Crawling Controversy
PerplexityBot indexes for citations; Perplexity-User fetches pages users ask about and generally ignores robots.txt. Plus the August 2025 Cloudflare report that Perplexity crawled sites that blocked it.
How to Verify a Real AI Bot (IP Ranges, Reverse DNS)
User-agent strings are spoofable, so verify AI bots by their published IP-range files, reverse DNS or request signatures, not by name. How to confirm a request really is GPTBot, ClaudeBot or PerplexityBot.
Can robots.txt Stop AI Scrapers?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Does Cloudflare Block AI Crawlers by Default?
For new domains, partly. Since Sep 15, 2026 new ad-supported domains refuse AI training and block agent bots on ad pages; search stays allowed. Existing domains keep their settings. Edge rules override robots.txt.
Why robots.txt Won't Block AI Bots (and What Actually Does)
Do AI crawlers respect robots.txt? The compliant ones do, user-triggered fetchers often don't, and scrapers ignore it. robots.txt asks; a WAF enforces. Here's the split, bot by bot.
Which AI Bots Should You Block? (And Why robots.txt Won't Stop Them)
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.