Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

How AI Crawlers Work: From Request to Model, Answer, or Visit

How AI crawlers fetch and use your content: the request, user-agent identification, the robots.txt check, why most don't run JavaScript, and the three destinations: model training, a citation index, or a live user.

By Andrej Ruckij · · 5 min read

How AI Crawlers Work: From Request to Model, Answer, or Visit

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: An AI crawler sends an HTTP request to your page, identifies itself with a user-agent string, checks robots.txt if it’s compliant, and reads the HTML it gets back. Most major AI crawlers don’t run JavaScript, so content that only appears after client-side rendering is invisible to them; render it on the server. What happens next depends on the bot’s job: a training crawler feeds the content into model weights, a search crawler indexes it to cite later, and a user-fetch agent hands it to a person who asked. Same fetch, three destinations.

Understanding the mechanics makes the access decision in block-or-allow-ai-crawlers much easier.

The basic request flow

Mechanically, an AI crawler behaves like any web client:

  1. It requests a URL with a standard HTTP GET.
  2. It identifies itself with a user-agent string, such as GPTBot/1.4 (OpenAI’s current version). Reputable bots use a documented token; see the crawler directory.
  3. A compliant bot checks robots.txt before fetching and honours Disallow rules. A non-compliant one skips it. User-triggered fetchers sit in between: several (Perplexity-User, Google-Agent) generally skip robots.txt because a person asked for the page, while Claude-User honours it.
  4. It reads the response. Most major AI crawlers fetch JavaScript files but don’t execute them. Vercel and MERJ’s December 2024 analysis found that none of the major AI crawlers rendered JavaScript, naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot; CCBot doesn’t render either. The exceptions they found were Gemini, which uses Googlebot’s rendering, and Applebot. So if your prices, product details or article text only appear after client-side rendering, most AI crawlers never see them. Server-side rendering or static HTML is what makes the content readable.

The three destinations

The fetched page goes to one of three places (glossary/ai-crawler):

  • Training. The content becomes training data and ends up in a model’s weights. No link back, no traffic. Examples: GPTBot, ClaudeBot, CCBot.
  • Search / retrieval. The content is indexed so the AI can cite it, with a link, when it answers a relevant question. Examples: OAI-SearchBot, PerplexityBot, Claude-SearchBot.
  • User-fetch. The content goes straight to a person who asked the AI to open that page. Examples: ChatGPT-User, Perplexity-User, Claude-User.

That’s why “block all AI” is a blunt instrument: it treats three different transactions as one.

How AI crawlers differ from search crawlers

Googlebot crawls to index for ranking, and historically traded crawling for referral traffic. AI training crawlers break that bargain: they fetch a lot and send little back (ai-crawler-traffic-impact has the numbers). AI search crawlers are closer to the old bargain, since they cite and link. So an AI crawler can be more extractive than Googlebot or roughly like it, depending on its job.

Identity is claimed, not proven

The user-agent string is self-reported and easy to fake. A scraper can call itself GPTBot and have nothing to do with OpenAI. That’s why verification uses the operator’s published IP ranges rather than the name, and why robots.txt, which keys off the claimed user agent, only works on bots that choose to be honest. Enforcement against the dishonest ones happens at the firewall (robots-txt-vs-waf-ai-bots).

For AI agents this is starting to change. Under Web Bot Auth, an agent signs each request with HTTP Message Signatures (RFC 9421), so its identity can be checked rather than taken on trust. ChatGPT agent already signs its requests (Signature-Agent: "https://chatgpt.com"), Google is experimenting with it for Google-Agent, and Cloudflare lists a bot as verified only if it passes Web Bot Auth signature checks or IP validation. See verify-ai-bots.

Key takeaways

  • An AI crawler requests a URL, identifies itself by user agent, checks robots.txt if compliant, then reads the HTML.
  • Most major AI crawlers don’t execute JavaScript; render important content on the server.
  • The same fetch feeds one of three destinations: training, search, or a live user.
  • Training crawlers break the search bargain; search crawlers keep it by citing and linking.
  • User-agent identity is claimed, not proven. Verify by IP range or Web Bot Auth signature, and enforce at the firewall.

Sources

#ai-crawlers#how-it-works#robots-txt#geo