Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

How to Block AI Scrapers: The Complete Enforcement Guide (2026)

robots.txt won't stop scrapers that ignore it. The enforcement layer, with working config: rate limiting, bot verification, WAF rules, IP and ASN blocking, and tarpits as a last resort.

By Andrej Ruckij · · 7 min read

How to Block AI Scrapers: The Complete Enforcement Guide (2026)

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: To stop a scraper that ignores robots.txt, you enforce at the edge. Work up a ladder: rate-limit everything, verify bots by IP range before trusting their names, block confirmed-bad traffic with WAF rules, block or challenge by IP and ASN when scrapers rotate, and use a tarpit only under sustained pressure. Compliant bots don’t need any of this; robots.txt handles them.

robots.txt asks and a firewall enforces; why that is, and which bots honour the file, is covered in robots-txt-vs-waf-ai-bots. This page is the how-to for everything robots.txt can’t reach.

First, the dividing line

There are two populations, and they need different tools:

  • Compliant bots (GPTBot, ClaudeBot, PerplexityBot and the like) read robots.txt and obey. Set your policy in block-or-allow-ai-crawlers and you’re done.
  • Non-compliant scrapers ignore robots.txt: the Bytespider type, anonymous scrapers, and crawlers that fake their identity. Everything below is for them.

The line isn’t always fixed by company. In August 2025 Cloudflare reported Perplexity crawling sites that had disallowed it, using an undeclared “Chrome on macOS” identity alongside its declared bot (Perplexity disputed this). When sites blocked the declared crawlers, the undeclared traffic still got through, and that is what needed enforcement. So “compliant bots need no enforcement” holds only as far as a company’s declared crawlers are the whole story (perplexity-crawlers).

The enforcement ladder

Most sites never need the top rungs.

1. Rate limiting (start here)

Cap requests per IP per time window. It targets behaviour rather than identity, so it catches aggressive and disguised crawlers alike, and it rarely touches real visitors, who don’t make hundreds of requests a minute. On nginx:

http {
    limit_req_zone $binary_remote_addr zone=perip:10m rate=2r/s;

    server {
        location / {
            limit_req zone=perip burst=20 nodelay;
            limit_req_status 429;
        }
    }
}

Each client IP gets an average of 2 requests per second with bursts up to 20; anything beyond the burst is rejected. nginx rejects with 503 by default, so limit_req_status 429 returns the more accurate “Too Many Requests”. Pick the rate from your own logs; a page that loads many assets from the same host needs a higher burst.

2. Verification (before you trust a name)

User-agent strings are spoofable, so confirm identity before you act on one: check the request IP against the operator’s published IP-range file, use forward-confirmed reverse DNS, or check a request signature. A request claiming to be GPTBot from outside OpenAI’s ranges is a scraper. Full method: verify-ai-bots.

3. WAF rules

This is the actual enforcement step: a rule returns a 403 at the edge before the request reaches your server. Cloudflare custom rules are available on every plan, including Free (5 rules per zone). Two examples in Cloudflare’s rules language:

Block a crawler you never want, by user-agent (action: Block):

lower(http.user_agent) contains "bytespider"

String comparisons in Cloudflare expressions are case-sensitive, which is why the header is lowercased first. A user-agent rule only catches bots that name themselves honestly; use it for those and rely on rates, IPs and ASNs for the rest.

Challenge traffic from networks a scraper runs from, while letting Cloudflare’s known good bots through (action: Managed Challenge):

ip.src.asnum in {64496 64497} and not cf.client.bot

Replace 64496 64497 (ASNs reserved for documentation) with the networks you see in your logs. Values in a set are separated by spaces, not commas.

4. IP and ASN blocking

An IP block stops one address. It’s precise and right for a verified-bad source, but a scraper drawing from a large cloud or residential IP pool can rotate addresses faster than you can list them.

An ASN (Autonomous System Number) identifies the network a range of IPs belongs to: a hosting provider, cloud platform or ISP. Acting on the ASN catches every address a scraper uses inside that network, which defeats rotation that stays within one host. It’s also blunt:

  • Good for scrapers running out of a datacenter network with no real visitors on it.
  • Dangerous on networks that also carry people, such as large ISPs and big clouds. There, challenge instead of hard-blocking, as in the rule above, and keep hard ASN blocks for networks that are clearly all bots.

5. Tarpits (last resort)

Under sustained pressure you can switch from blocking to wasting the crawler’s resources with Nepenthes, Anubis or Cloudflare AI Labyrinth. One warning if you reach for Anubis: its default policy blocks every AI-associated user agent, including AI search crawlers and user-fetch bots such as OAI-SearchBot and ChatGPT-User. If you want to stay visible in AI search, switch to its ai-block-moderate policy. Tradeoffs for all three: ai-crawler-tarpits.

The managed shortcut

CDNs bundle most of this ladder: bot scoring, verified-bot lists, ASN data, rate rules and a decoy maze. For most teams, enabling and tuning a managed AI-bot setting beats hand-written rules. Know what the defaults are, though. On Cloudflare, AI-crawler blocking became the default for new domains at sign-up in July 2025, with an opt-in toggle for existing ones. Since September 15, 2026, new ad-supported domains on Cloudflare refuse AI training by default (training-only bots such as GPTBot and ClaudeBot are blocked; Googlebot, Bingbot and Applebot stay in search) and block agent bots on pages with ads; search bots stay allowed. Existing domains change only if someone switches settings, and Cloudflare’s plain “Block” settings now reach Googlebot, Applebot and Bingbot too (does-cloudflare-block-ai-crawlers). Because a CDN rule sits at the edge, it overrides your robots.txt, so check the two agree.

A realistic expectation

Enforcement makes crawling you expensive and slow, not impossible. A determined scraper on a large residential proxy pool is hard to stop completely; the goal is to push its cost above what it gets from you. And none of this should touch the bots you want. Blocking a search bot means losing the documented route into that assistant’s answers (does-blocking-ai-bots-hurt-seo), so verify and allow the search and user-fetch crawlers that cite you.

Key takeaways

  • robots.txt handles compliant bots; this ladder handles the rest.
  • Rate-limit first: it targets behaviour, catches disguised crawlers, and rarely hits real visitors.
  • Verify by IP range before trusting a name; user-agent rules only catch honest bots.
  • IP blocks lose to rotation; ASN rules catch rotation within a network but must be challenges, not blocks, where real users share the network.
  • If you use Anubis and want AI search visibility, switch from its default policy to ai-block-moderate.

Sources

#ai-crawlers#enforcement#waf#ip-blocking#asn#rate-limiting#bot-management