Skip to content
Primores AI wiki

WAF (Web Application Firewall) — Definition

WAF (Web Application Firewall)

TL;DR: A WAF is a firewall that inspects incoming web requests and blocks unwanted ones at the edge — before they reach your server. For AI bots, it’s the enforcement layer: it actually blocks, whereas robots.txt only asks.

What it means

A web application firewall sits in front of your website (usually at your CDN or host) and filters HTTP requests against a set of rules. A request that matches a block rule gets rejected — typically a 403 Forbidden — before it ever touches your origin server. Unlike robots.txt, a WAF does not depend on the visitor’s cooperation: it imposes the decision on every request, compliant or not. That is exactly what makes it the real control point for AI bot access.

Why it matters

robots.txt governs only the bots that choose to obey it; a WAF governs all of them. This produces the single most important rule in AI bot access control: if a WAF rule and your robots.txt disagree, the WAF wins. A managed “block all AI crawlers” firewall rule will keep out OAI-SearchBot even if your robots.txt says Allow — because the firewall acts at the edge and never reads robots.txt. So if you’ve enabled any default AI-blocking at your CDN, reconcile it against your robots.txt intentions, or you may be invisible to AI search without realizing it.

How it works / examples

  • Compliant bots (glossary/gptbot, ClaudeBot): robots.txt is enough; a WAF is optional.
  • Non-compliant scrapers (glossary/bytespider, stealth crawlers that spoof user-agents): only a WAF + IP/ASN rules actually stop them.
  • Verification: because user-agent strings are fakeable, strong WAF rules verify against the operator’s published IP ranges, not the bot’s self-reported name.

Cloudflare’s AI Crawl Control is a managed WAF-layer example; it can block AI crawlers by default and even charge them via glossary/pay-per-crawl.

Sources