Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

How to Verify a Real AI Bot (IP Ranges, Reverse DNS)

User-agent strings are spoofable, so verify AI bots by their published IP-range files, reverse DNS or request signatures, not by name. How to confirm a request really is GPTBot, ClaudeBot or PerplexityBot.

By Andrej Ruckij · · 6 min read

How to Verify a Real AI Bot (IP Ranges, Reverse DNS)

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: A bot’s user-agent is self-reported and easy to fake, so never trust the name alone. Check the request IP against the operator’s published IP-range file (OpenAI, Anthropic, Perplexity, Google, Apple, Amazon and Common Crawl publish them), use forward-confirmed reverse DNS where the operator documents a hostname, or check the cryptographic request signature that some AI agents now send. A request calling itself GPTBot from outside OpenAI’s ranges is not GPTBot.

Verification sits under every other step of the enforcement ladder: allow and block rules are only as trustworthy as your ability to confirm who is asking.

Why the user-agent isn’t enough

The user-agent is an HTTP header the client sets itself. Any scraper can send GPTBot while having nothing to do with OpenAI, and disguised crawlers do exactly this to slip past name-based rules (see the Perplexity stealth-crawl dispute). A robots.txt rule or firewall rule keyed on the name catches honest bots and misses the dishonest ones.

Method 1: published IP-range files (best)

Most major AI operators publish machine-readable lists of the IP ranges their bots use. Check the request IP against the right file:

OperatorIP-range file
OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot)openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, openai.com/adsbot.json
Anthropic (ClaudeBot, Claude-SearchBot, Claude-User)claude.com/crawling/bots.json (one list for all three)
Perplexity (PerplexityBot, Perplexity-User)perplexity.ai/perplexitybot.json, perplexity.ai/perplexity-user.json
Google (Googlebot and other common crawlers, including Google-CloudVertexBot; user-triggered fetchers; Google-Agent)common-crawlers.json, user-triggered-fetchers.json, user-triggered-agents.json under developers.google.com/static/crawling/ipranges/
Apple (Applebot)search.developer.apple.com/applebot.json
Amazon (Amazonbot, Amzn-SearchBot, Amzn-User)separate lists under developer.amazon.com/amazonbot/ (ip-addresses/, searchbot-ip-addresses/, live-ip-addresses/)
Common Crawl (CCBot)index.commoncrawl.org/ccbot.json

Two caveats. Anthropic’s file is a single unlabelled list, so it tells you an IP belongs to Anthropic but not which of its three bots it is, and Anthropic itself says IP blocking may not reliably guarantee an opt-out; use robots.txt for that. And Google’s common-crawlers file also covers Google-CloudVertexBot, which crawls when a site owner builds a Vertex AI agent and doesn’t cite you anywhere, so an IP match there doesn’t mean search traffic.

If a request carries a bot’s user-agent but its IP isn’t in that operator’s range, it isn’t the real bot. Full user-agent reference: ai-crawler-user-agents-directory.

Method 2: reverse DNS

Some operators document a hostname their crawlers resolve to, and a forward-confirmed reverse DNS check uses it: look up the hostname for the request IP, confirm it sits under the expected domain, then resolve that hostname forward and confirm it returns the same IP. Googlebot hosts resolve under googlebot.com, Applebot under applebot.apple.com, and Common Crawl’s CCBot (IPv4) under crawl.commoncrawl.org. It’s more work than an IP lookup but defeats simple spoofing.

Where an operator publishes neither ranges nor a hostname, you can’t verify by network at all. Meta is the main example: its crawler documentation lists Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer and FacebookExternalHit but publishes no IP ranges. For those you fall back on behaviour (Method 4).

Method 3: request signatures (Web Bot Auth)

AI agents that browse and act for a user increasingly prove who they are cryptographically. Under Web Bot Auth, the agent signs each request with HTTP Message Signatures (RFC 9421) and points to a public key directory, so you check the signature instead of trusting the header. ChatGPT agent signs every request with Signature-Agent: "https://chatgpt.com", and Google is experimenting with Web Bot Auth for Google-Agent (https://agent.bot.goog). Cloudflare now requires either Web Bot Auth or IP validation before it lists a bot as verified. A valid signature is stronger evidence than an IP match, because a scraper on the same cloud network can’t borrow it.

Method 4: behavioural signals (the backstop)

When identity can’t be confirmed, behaviour gives it away: request rates far above a human, sequential crawling of deep URLs, ignoring robots.txt, rotating user-agents or IPs within a session. These don’t prove who a bot is, but they show something automated and uncooperative is happening, which is enough to rate-limit or challenge it.

Putting it together

  • For bots you allow (search and user-fetch), verify by IP range so a scraper can’t borrow their names to get past your other rules.
  • For bots you block, robots.txt handles the honest ones; rate limits, IP and ASN rules handle the rest (how-to-block-ai-scrapers).
  • CDNs like Cloudflare keep verified-bot lists and do this checking for you (does-cloudflare-block-ai-crawlers), and they have removed operators caught disguising their crawlers.

Key takeaways

  • Never trust the user-agent alone; it’s spoofable.
  • Check the IP-range file first. OpenAI, Anthropic, Perplexity (on perplexity.ai), Google, Apple, Amazon and Common Crawl publish them.
  • Anthropic’s list doesn’t say which bot an IP belongs to; Google’s common-crawlers list includes Google-CloudVertexBot.
  • Use forward-confirmed reverse DNS where a hostname is documented. Meta publishes neither, so its bots can only be judged by behaviour.
  • For AI agents, check the Web Bot Auth signature where one is sent.

Sources

#ai-crawlers#verification#ip-ranges#waf#bot-management