Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

AI Crawler Regulation in the EU and UK (2026): What Site Owners Should Know

The EU AI Act gives machine-readable opt-outs legal weight for AI training, and robots.txt is the one the GPAI Code names; the UK dropped its text-and-data-mining opt-out plan in March 2026 and is waiting. What that means for your robots.txt.

By Andrej Ruckij · · 7 min read

AI Crawler Regulation in the EU and UK (2026): What Site Owners Should Know

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: In the EU, the AI Act requires general-purpose AI providers to respect machine-readable rights reservations, which gives machine-readable opt-outs legal weight for training; robots.txt is the protocol the voluntary GPAI Code of Practice commits signatories to respect. Since 2 August 2026 the Commission’s AI Office can fine providers that don’t comply, but models placed on the market before 2 August 2025 have until 2 August 2027 to comply, and the Commission is still working on the list of accepted protocols. In the UK, the government dropped its proposed text-and-data-mining exception with opt-out in March 2026, left an AI bill out of the May 2026 King’s Speech, and is leaning on industry licensing. Net: your robots.txt is starting to carry legal weight in the EU; UK law is unsettled.

A cluster under the block-vs-allow tradeoff. This is fast-moving policy, so treat the specifics as a late-2026 snapshot, not settled law.

The EU’s framework links copyright and the AI Act in a way that matters directly for site owners:

  • Under the EU’s text-and-data-mining rules, rightsholders can reserve their rights from TDM, and the reservation must be expressed in a machine-readable way.
  • The AI Act (Art. 53) commits general-purpose AI providers to put in place a copyright policy that identifies and complies with those machine-readable rights reservations. The obligation has applied since 2 August 2025. Since 2 August 2026 the Commission’s AI Office can enforce it, with fines of up to €15 million or 3% of global annual turnover, whichever is higher. The Digital Omnibus on AI (Regulation (EU) 2026/1744, in force 27 July 2026) postponed the high-risk rules but did not delay this GPAI enforcement date.
  • The fining power reaches less than it sounds. Under Art. 111(3), providers of GPAI models “placed on the market before 2 August 2025” have until 2 August 2027 to comply. For those models, the date that matters is August 2027.
  • The voluntary GPAI Code of Practice (July 2025) has signatories commit to honouring robots.txt and other appropriate machine-readable rights-reservation protocols.
  • The European Commission ran a consultation (1 December 2025 to 23 January 2026) to identify which opt-out protocols are generally agreed, and has said it will publish that list and review it at least every two years. No final list had been located as of October 2026 (unverified as of Oct 2026).

The practical upshot: in the EU, a machine-readable signal like robots.txt (and the protocols emerging next to it) is moving from polite request toward legally relevant opt-out. Compliant AI providers have a regulatory reason, not just etiquette, to honour it for training. That doesn’t make robots.txt an enforcement mechanism (see robots-txt-vs-waf-ai-bots), but it raises the stakes for providers who ignore it.

The UK: the opt-out plan was dropped

The UK took the opposite turn in 2026:

  • The government’s December 2024 consultation on copyright and AI (it ran to 25 February 2025) floated an EU-style commercial TDM exception with an opt-out for rightsholders.
  • After widespread opposition, notably from the creative industries, the government’s March 2026 report dropped that as the preferred option.
  • The UK is now waiting: letting industry-led licensing develop, watching other countries, and deferring legislation.

So in the UK there’s currently no new AI-training copyright exception and no statutory opt-out regime. The King’s Speech of 13 May 2026 included no AI bill, and in March the Financial Times reported that the comprehensive AI bill had been pushed to 2027. The government’s stated further work covers transparency and creator control, labelling AI content, digital replicas, and licensing support for smaller creators.

What this means for your robots.txt

  • If you serve EU audiences and want to opt out of AI training: set your training-bot disallows deliberately. The bot names are in ai-crawler-user-agents-directory and the full file is in block-or-allow-ai-crawlers. Under the EU framework, a machine-readable reservation is the recognised way to express the opt-out, and compliant providers are expected to respect it.
  • Don’t mistake legal weight for enforcement. Regulation pressures compliant providers; non-compliant scrapers still need a firewall (robots-txt-vs-waf-ai-bots). Law and enforcement are different layers.
  • Keep the training/search distinction. Opting out of training (the regulated use) doesn’t require blocking search bots. You can stay visible in AI search while reserving training rights.
  • Expect change. The EU is still settling the protocol list and the UK may revisit. Check your policy again as the rules settle.

Key takeaways

  • EU: the AI Act and the TDM rules give machine-readable opt-outs legal weight for AI training; robots.txt’s place rests on the voluntary GPAI Code of Practice until the Commission publishes its list of agreed protocols. The AI Office can fine GPAI providers since 2 August 2026, but models on the market before 2 August 2025 have until 2 August 2027 to comply.
  • UK: dropped the TDM-exception-with-opt-out plan in March 2026, no AI bill in the May 2026 King’s Speech, industry-led licensing for now.
  • Your robots.txt is gaining legal weight in the EU, but regulation binds compliant providers, not scrapers. Enforcement is still a firewall job.
  • You can opt out of training without giving up AI search visibility.

Sources

#ai-crawlers#regulation#eu-ai-act#copyright#robots-txt