Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

Do AI Crawlers Cost You Money? Bandwidth, Server Load, and the Broken Bargain

Every AI crawl is a request your server pays to answer, and training crawlers send few visitors back. The crawl-to-referral numbers for 2025 and 2026, how to measure your own exposure, and when the cost justifies blocking.

By Andrej Ruckij · · 6 min read

Do AI Crawlers Cost You Money? Bandwidth, Server Load, and the Broken Bargain

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: Every crawl is a request your server pays to answer; how much that costs depends on your site, and nobody publishes a good general figure. What is measured is the other side of the deal: training crawlers send very few visitors back. In Cloudflare Radar’s July 2026 figures, Anthropic crawled about 1,917 pages for every visitor it referred, OpenAI about 251 and Perplexity about 289, against 4.7 for Google. The gap shrank a lot for Anthropic and OpenAI since mid-2025 but grew for Perplexity, so it isn’t closing across the board. For a small cached site the cost is usually negligible; for a large or dynamic one it can be a real line item. Measure it in your own logs before deciding.

Most coverage treats AI crawling as a content question. There’s also a plain cost question, and it feeds into the block-or-allow decision.

The costs AI crawling imposes

When a crawler fetches your pages at volume, you pay in a few ways:

  • Bandwidth and egress. Every fetched page is data served, which costs money on metered hosting and CDNs.
  • Server and origin load. Heavy crawling uses CPU and database time, especially on uncached dynamic pages, and can slow the site for real users.
  • Cache churn. Bots hitting deep, rarely visited URLs miss the cache and hit the origin.

On a small static site this rarely matters. On a large catalogue, a dynamic app or a site near capacity, it can.

The asymmetry

AI crawling became a cost grievance, not just a copyright one, because the old exchange broke. Search engines crawled you and sent traffic back. AI crawlers mostly don’t. Cloudflare’s first measurement, published in July 2025 for the week of June 19 to 26, put Anthropic at about 71,000 pages crawled per referral. Radar’s full-month figures for July 2025 had Anthropic at 38,744 and OpenAI at 1,104. You pay to serve the pages, and the referral that used to pay you back mostly isn’t there.

Where the ratios went

Cloudflare Radar tracks the crawl-to-refer ratio continuously. The full-month figures below were pulled from Radar by TechnologyChecker and compiled by eCorp IT:

OperatorLate Jun 2025 (Cloudflare)Jul 2025 (Radar)Jul 2026 (Radar)
Anthropic~71,000:138,744:11,917:1
OpenAIn/a1,104:1251:1
Perplexityn/a195:1289:1
Microsoftn/a40.4:136.2:1
Googlen/a5.5:14.7:1

Anthropic’s and OpenAI’s ratios fell sharply. Perplexity’s went the other way, from 195 to 289, and TechnologyChecker’s later pull put it at 1,083 for the third quarter of 2026. Microsoft and Google barely moved. Read the table as a direction, not a rate: the ratios swing from month to month (Anthropic was at 3,386 in June 2026 and 988 for the third quarter), and Cloudflare notes that native apps such as Claude’s send no Referer header, so every AI ratio overstates the gap by an unknown amount.

Training traffic is also growing as a share of crawling: Cloudflare says 52% of crawler requests were for AI training in June 2026, up from 22% in spring 2025. OpenAI’s ratio has risen again since July (425:1 in August, 354:1 in a provisional September reading, per the same eCorp IT summary of Radar). My read: the gap for Anthropic is much smaller than in 2025, OpenAI’s is moving around, and AI operators as a group still take far more per visitor sent than Google does.

This lands mainly on training crawlers. Search crawlers are the ones that produce citations and referrals.

How to measure your own exposure

  1. Check your server logs or analytics for AI user agents (GPTBot, ClaudeBot, CCBot, Bytespider and others) and tally requests and bytes served.
  2. Compare with referral traffic from AI sources. Are the crawlers sending anyone back?
  3. Watch for spikes. Aggressive or buggy crawlers, and non-compliant ones like Bytespider, can crawl in bursts that dwarf normal bot traffic.
  4. Look at your CDN. Cloudflare and others report AI-bot traffic in their dashboards, and Cloudflare Radar publishes the crawl-to-refer ratio per operator as a benchmark for your own logs.

When the cost justifies blocking

  • High crawl volume, almost no referrals, real infrastructure cost: a strong case to block the training crawlers responsible, and to rate-limit or firewall the non-compliant ones.
  • On Cloudflare, block training without blocking search. Its plain Block settings now also reach mixed-use crawlers (Googlebot, Applebot, Bingbot); use Disallow AI Training if bandwidth is the reason and search still matters (does-cloudflare-block-ai-crawlers).
  • Modest volume on a cached static site: rarely worth acting on for cost alone. Decide on the content and visibility merits instead.
  • Don’t block search crawlers to save bandwidth. They’re the ones that cite you (does-blocking-ai-bots-hurt-seo).

For crawlers that ignore robots.txt and drive cost, the fix is enforcement at the firewall, not a directive (robots-txt-vs-waf-ai-bots).

Key takeaways

  • AI crawlers cost bandwidth and server load, from negligible on small static sites to meaningful on large dynamic ones.
  • In July 2026, Anthropic crawled about 1,917 pages per referral, OpenAI 251 and Perplexity 289, against Google’s 4.7.
  • The gap shrank for Anthropic and OpenAI since 2025 but grew for Perplexity.
  • It’s mainly a training-crawler cost; search crawlers still send referrals.
  • Measure in your logs or CDN first; block costly training and non-compliant bots, not search bots.

Sources

#ai-crawlers#bandwidth#server-load#robots-txt#performance