Why are publishers blocking AI crawlers, and should your business do the same?

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: Big news publishers block AI bots because their articles are the product, AI crawlers take far more than they send back, and a blocked site has something to sell. In BuzzStream’s January 2026 count, 79% of the top UK and US news sites block at least one AI training bot and 71% block at least one AI search or live-fetch bot. The selling works for a few: News Corp’s OpenAI deal is reported at $250 million over five years, Meta reportedly pays it up to $50 million a year, Amazon reportedly pays the New York Times $20-25 million a year. Blocking also costs something, though less than early reports said: in the current version of a Rutgers and Wharton study, newspaper publishers that blocked lost about 7% of traffic in the six weeks after, an effect that loses statistical significance after about 20 weeks. For a business whose website sells something else, that playbook mostly doesn’t fit, and the 2026 evidence hasn’t changed my answer. My rule: if your content is what you sell, block training and explore a paid route; if your content sells a product or service, allow AI search and live-fetch bots and treat training as a minor call. Never block the bots that fetch a page because a person asked.

This piece sits under the block-vs-allow tradeoff. The publisher wave is the most visible AI-blocking story, and the most copied template. Below is what publishers actually do, what it got them, and how to tell whether any of it applies to you.

Who blocks, in numbers

The headline “80% of news sites block AI” describes the biggest brands. The wider base is lower, and outside news it’s much lower.

Study	Date	Sample	Finding
Reuters Institute (Richard Fletcher)	Feb 2024, end-2023 data	15 most-used news sites in each of 10 countries	48% blocked OpenAI’s crawlers, 24% Google’s AI crawler; from 79% in the US to 20% in Mexico and Poland
BuzzStream, via Press Gazette	Jan 2026	about 100 largest UK and US news sites	79% block at least one training bot, 71% at least one retrieval or live-search bot
Elmo news tracker	Sep 2026	1,156 news outlets	54.2% block at least one of GPTBot, Google-Extended and CCBot; GPTBot alone 49.7%
Elmo census	Sep 7, 2026	top 5,000 domains, all types	368 of the 2,771 sites with a robots.txt (13.3%) block at least one AI search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot); that’s 7.4% of all 5,000

A few details from those studies matter more than the averages:

A third of the top block everything. In BuzzStream’s top 50, 34% block every bot it checked, including the BBC, the New York Times, the Daily Mail, the Telegraph, AP News and the Wall Street Journal. 14% allow all of them, among them Fox News, The Independent, Politico and Substack.
The US blocks harder than the UK. 81% of US sites block a training bot against 77% in the UK; for Google-Extended it’s 58% against 29%.
Allowlists are the new default at the top. Search Engine Journal, citing Digiday, reported in June 2026 that Reuters and Time switched in May to blocking AI bots by default and letting in only approved crawlers, as People Inc. and The Atlantic had done earlier. After People Inc. made that switch, the number of user agents it blocks rose from about 2,100 to more than 30,000.
Few have reversed. The Reuters Institute found that during 2023 “none of the websites we examined had reversed their decision after deciding to block.” I found no named major publisher that has since unblocked.

(Updated Oct 2026, still true: an earlier version said publishers blocking training “often allow” search bots. The 71% figure shows most of the biggest block both.)

Why publishers block

The articles are the product. A news site sells its reporting through subscriptions and ads. An AI answer that summarises the story removes the reason to visit. The Telegraph’s SEO director, Harry Clarkson-Bennett, put it to Press Gazette this way: “Publishers are blocking AI bots using the robots.txt because there’s almost no value exchange.”

The crawl-for-traffic exchange broke. Cloudflare Radar’s crawl-to-referral figures for July 2026, as pulled by TechnologyChecker, show Anthropic crawling about 1,917 pages for every visit it referred, OpenAI 251 and Perplexity 289, against 4.7 for Google. A year earlier Anthropic was at 38,744 and OpenAI at 1,104, so the ratio shrank for those two, while Perplexity’s grew from 195. Cloudflare has noted that traffic referred by Claude’s native app carries no referrer header, so Anthropic’s figure in particular overstates the gap. TollBit, which sells bot monetisation to publishers, reported that click-through from AI apps to publisher sites fell from 0.8% in the second quarter of 2025 to 0.27% in the fourth, while AI bot visits rose from one per 200 human visits to one per 31. More on the traffic side: ai-crawler-traffic-impact.

Blocking is a bargaining position. People Inc.’s chief innovation officer, Jonathan Roberts, told AdExchanger in May 2026: “We’re blocking bots so that we can permission them and so we can have control over who has accesses and how.” People Inc. lets 38 crawlers in and blocks 30,000 to 35,000 a day. A blocked publisher can sell access. An open one is giving it away.

What the deals are worth

Most deal values have never been disclosed, so any total is a floor. These are the reported figures, all from press reports rather than company filings unless noted:

Deal	Reported value	What’s covered
OpenAI and News Corp (2024)	$250 million over five years	News Corp content
Meta and News Corp (Mar 2026, per the WSJ)	up to $50 million a year, at least three years	current content for Meta’s AI products, archives for training
Amazon and the New York Times (2025, per the WSJ)	$20-25 million a year	training, plus summaries and excerpts in Alexa; includes The Athletic and NYT Cooking
OpenAI and Reddit (2024)	about $70 million a year	Reddit data
Google and Reddit (2024)	$60 million a year	Reddit data; Reddit’s IPO filing disclosed $203 million in total data-licensing contract value
OpenAI and Dotdash Meredith (2024)	about $16 million a year	content licence
OpenAI and the Financial Times (2024)	$5-10 million a year	content licence
OpenAI and Axel Springer (2023)	“tens of millions of euros”	content licence
Perplexity Publishers Program (2024)	$42.5 million revenue-share pool	shared across participants

The pattern is obvious once it’s in a table. The buyers pay for scale, brand and archives: national newspapers, a giant forum, a big magazine group. Microsoft’s Publisher Content Marketplace, which pays publishers when their content informs an AI answer, had eight members in May 2026. I found no reported licensing deal with a mid-sized business website, and no published figure for what a small site could earn.

What blocking cost the publishers who did it

Evidence points both ways, so here’s both.

Traffic dipped, modestly. Hangcheng Zhao (Rutgers) and Ron Berman (Wharton) compared publishers’ traffic before and after they blocked AI crawlers. Their core sample is 30 newspaper publishers, extended to the top 500 news domains in the Semrush and Comscore checks, and the main analysis covers the period before May 2024. In the current version of the paper (v4, April 2026), traffic fell about 7.1% in the six weeks after blocking in SimilarWeb data (significant at the 1% level), about 6.7% in Semrush (significant only at 10%) and about 6.3% in Comscore (not significant). Mid-sized publishers showed a smaller or similar dip that wasn’t significant either. Over a longer window the estimates stay negative but lose significance after about 20 weeks. PPC Land reported much larger figures (23.1% and 13.9%) from an earlier version of the paper; the authors no longer report those. The authors’ explanation is lost exposure: once blocked, the AI stops naming the publisher as a source. SimilarWeb gives estimates, Comscore covers only US desktop, and the paper can’t fully rule out other site changes made at the same time, though most blocking sites changed nothing else in robots.txt.

Citations mostly didn’t stop. BuzzStream analysed about 4 million AI citations from 3,600 prompts across ChatGPT, Gemini, AI Overviews and AI Mode (last updated April 2026). Of the news sites blocking OpenAI’s search bot, OAI-SearchBot, 82.4% were still cited by ChatGPT; for ChatGPT-User the figure was 70.6%, and for the training bot GPTBot 88.2%. That sits awkwardly next to OpenAI’s own documentation, which says: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.” BuzzStream’s author offers guesses (Common Crawl copies, bots ignoring robots.txt, answers built from search-result snippets alone) and admits “we don’t know much about how this stuff works yet.”

Bots don’t all obey. TollBit found that 30% of AI bot scrapes didn’t comply with explicit robots.txt permissions (via Search Engine Journal). A robots.txt line is a request; enforcement happens at the firewall. See robots-txt-vs-waf-ai-bots.

One small site went the other way. Pixelmojo, an AI product studio, unblocked CCBot, Google-Extended, anthropic-ai and Applebot-Extended on May 16, 2026. Its commit message: “being known to future model generations mattered more to us than blanket protection.” It also withdrew an earlier claim that blocking had raised its citations, because it had no before-and-after data. It’s one site, self-reported and unmeasured, but it’s the reasoning I’d expect most marketing sites to land on.

What I take from this: for a news publisher, robots.txt blocking costs a small, short-lived dip in traffic and buys less protection than people assume. For you, the same evidence cuts two ways. Blocking AI search bots may cost less visibility than feared, because you can still get cited. It also protects less than you’d hope. Either way the case for blocking a marketing site gets weaker, not stronger.

The new ways to charge bots

Blocking is turning into pricing. Four options, all early:

Option	How it works	Cost to the site	Status (Nov 2026)
Cloudflare pay per crawl	You set a price per domain; a crawler that hasn’t agreed to pay gets an HTTP 402 “Payment Required” with the price, one that has gets the page; Cloudflare is merchant of record	fees not published on the docs page	closed beta, by request; Enterprise customers via their account executive
TollBit	You configure your CDN, WAF or bot tool to route identified AI agents to a TollBit-managed access point; terms and rates are applied at the moment of access	”free for publishers to use”, setup “typically takes less than 30 minutes”	live; TollBit’s page doesn’t name its fee structure
RSL (Really Simple Licensing) 1.0	Machine-readable licence terms on top of robots.txt: usage categories `ai-all`, `ai-input`, `ai-index`, plus a “contribution” payment option	free standard	official since Dec 10, 2025; backed by Yahoo, Ziff Davis, O’Reilly, Cloudflare, Akamai, Stack Overflow, Creative Commons; the announcement named no AI company that has agreed to honour it
Microsoft Publisher Content Marketplace	Publishers are paid when their content informs an AI answer	by application	eight members in May 2026

Stack Overflow is the clearest named user of pay per crawl. In a February 2026 conversation on its blog, its team said it runs pay per crawl alongside its larger licensing deals, as a pay-per-use option for bots that wouldn’t sign a full contract, after what one engineer called “whack-a-mole” blocking of individual bots. They published no prices or revenue. I found no published payout figures for any of these systems, so treat them as experiments.

The lawsuits that matter

Courts haven’t settled whether training on published content is fair use, and the answers so far point in different directions:

New York Times v. OpenAI and Microsoft (filed Dec 27, 2023, Southern District of New York). Judge Sidney Stein largely denied OpenAI’s motion to dismiss in March 2025. OpenAI was ordered to produce 20 million ChatGPT logs. The US Justice Department filed a statement backing OpenAI on September 2, 2026. Both sides moved for summary judgment on fair use on September 4; replies are due around November 6, 2026. No trial date is set.
Bartz v. Anthropic (books). In June 2025 the court held that training on the books was fair use, but keeping millions of pirated copies raised separate issues. Anthropic settled for $1.5 billion, which Wikipedia describes as the largest known settlement in a US copyright case; final approval came on July 20, 2026.
Thomson Reuters v. Ross Intelligence. On September 29, 2026 the Third Circuit affirmed that Westlaw’s headnotes are copyrightable and that Ross’s use of them to train its AI tool “was not fair use.” How far that reaches beyond this case is open.
Still active, per AI Lawsuit Tracker (Oct 30, 2026): Dow Jones and the New York Post v. Perplexity, Ziff Davis v. OpenAI, Britannica and Merriam-Webster v. Perplexity, the New York Daily News group v. OpenAI, Advance Local and others v. Cohere. Penske Media’s suit against Google over AI Overviews is listed as dismissed.

For most businesses the practical point is narrow. A clear opt-out in robots.txt is a written record of what you wanted; the EU and UK rules on training opt-outs are in ai-crawler-regulation-eu-uk. None of these cases needs you to block search bots to protect your rights.

What changed in autumn 2026

Cloudflare rebuilt its AI controls on September 15. Each domain now has separate Search, Training and Agent controls, and new ad-supported domains refuse AI training by default (training-only bots such as GPTBot and ClaudeBot are blocked; Googlebot, Bingbot and Applebot stay in search) and block agent bots on pages with ads, while search stays allowed. (Updated Nov 2026: an earlier version called the default the softer option; “Disallow AI Training” still blocks GPTBot, ClaudeBot and the other training-only crawlers, and only spares the mixed-use search crawlers.) Details: does-cloudflare-block-ai-crawlers.

A shared training opt-out arrived on October 30. An IETF draft by Kevin Kelley of Anthropic defines one robots.txt token, __usage_ai-training, that tells every participating AI company not to train on your content. It’s an individual draft with no formal IETF standing yet. Google says it aims to support it “in 2027” and will read it as a Google-Extended opt-out; Bing will honour it from February 1, 2027 and says it doesn’t change eligibility for Bing search results; a report on fireup.pro says OpenAI also committed to honouring it. One catch from the draft: User-Agent: * does not count. The token has to be named.

How to decide, by business type

The question that decides it: is your content the thing you sell, or does it sell something else?

Your business	Training bots	AI search bots	Live-fetch bots (a person asked)	Why
News, research, reference or course content with a large archive	block	your call; the biggest publishers block, but Zhao and Berman found about 7% less traffic in the first six weeks	allow unless you’re negotiating	your content is the product; consider TollBit or a pay-per-crawl waitlist
Small or mid publisher, niche blog with ads	block, or Cloudflare’s Disallow AI Training	allow	allow	no licensing buyer is coming; AI search is a source of readers
B2B, SaaS, agency, professional services	allow or block, minor either way	allow	allow	being named when a buyer asks AI about your category is the point
Online store	allow or block	allow	allow, these are shoppers	AI shopping answers need your product pages; Roberts noted big retailers like Walmart may still block scrapers so competitors can’t copy their catalogue, which matters at Walmart’s scale, not yours
Local business	allow	allow	allow	nothing to protect, everything to gain
Paywalled or member content	block	allow on public pages only	allow on public pages	the paywall protects the product; the public pages market it

My read on the training column for marketing sites: blocking training is cheap and does no harm to search, so it’s fine if it makes you comfortable. But there’s a real argument for allowing it, which is Pixelmojo’s: a model that learned about you is a model that can mention you without searching. Pick one and stop worrying about it.

This rule doesn’t fit you if you’re a large publisher already in talks with an AI company. Then blocking everything first, as Reuters, Time and People Inc. do, is a negotiating move and a lawyer should be in the room.

How to set it up

1. Check what you have now (5 minutes). Open https://<your-domain>/robots.txt in a browser. Look for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, CCBot, Google-Extended and any User-agent: * group with Disallow: /. A wildcard block hits every bot that follows robots.txt, search bots included.

2. Write the split (10 minutes). For a marketing site that wants to opt out of training and stay visible in AI answers: Disallow: / for the training bots (GPTBot, ClaudeBot, CCBot, Applebot-Extended, Google-Extended, plus the shared __usage_ai-training token), and Allow: / for the search and user-fetch bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User). The full file, with the Google-Extended caveat for Gemini citations, is in block-or-allow-ai-crawlers. Anthropic says the rules have to be in the robots.txt of each subdomain.

3. If you’re on Cloudflare, set the three controls (5 minutes). In the dashboard, open your domain, then Security Settings, and find the AI crawler controls. Set Search to Allow, Agent to Allow, and Training to Disallow AI Training if you’re opting out. Avoid Block on Training unless you’re willing to lose Googlebot and Bingbot crawling as well.

4. Check that the server agrees with the file (10 minutes). robots.txt can say “allow” while your firewall returns 403. From a terminal, compare a normal request with one sent as a bot:

curl -s -o /dev/null -w "%{http_code}\n" https://<your-domain>/
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://<your-domain>/
curl -s -o /dev/null -w "%{http_code}\n" -A "PerplexityBot" https://<your-domain>/

All should return 200 if you meant to allow them. A 403 means a firewall, CDN or hosting rule is blocking them regardless of robots.txt. Real bots also come from published IP ranges, so a user-agent test is a first check, not proof; verify-ai-bots covers the full method.

5. Check again in a month. Look in your server logs or Cloudflare’s AI crawler report for OAI-SearchBot, Claude-SearchBot and PerplexityBot requests. If they’re missing, something upstream is still blocking them.

Where I’m unsure

The traffic study’s main analysis covers newspaper publishers before May 2024, and its effects are small and fade. I found no equivalent study for business or ecommerce sites that blocked AI search bots.
BuzzStream’s citation finding and OpenAI’s documentation disagree, and nobody has a confirmed explanation yet. If blocking OAI-SearchBot really did nothing, the cost side of my advice shrinks; the benefit side shrinks too.
Crawl-to-referral ratios differ a lot by month and by source. The direction (AI takes far more than Google) is consistent; the exact multiples aren’t.

block-or-allow-ai-crawlers, the decision rule and the robots.txt template
does-blocking-ai-bots-hurt-seo, the SEO side of the same decision
ai-crawler-regulation-eu-uk, the legal backdrop to the licensing fights
ai-crawler-user-agents-directory, every bot name
does-cloudflare-block-ai-crawlers, Cloudflare’s defaults and controls
glossary/ccbot, Common Crawl, a focal point of publisher disputes