#robots-txt
21 posts tagged with robots-txt.
AI Crawler Access Checklist: 8 Steps for Site Owners
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
AI Crawler Access Control: The Complete Guide (2026)
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
AI Crawler Regulation in the EU and UK (2026): What Site Owners Should Know
The EU AI Act gives machine-readable opt-outs legal weight for AI training, and robots.txt is the one the GPAI Code names; the UK dropped its text-and-data-mining opt-out plan in March 2026 and is waiting. What that means for your robots.txt.
Do AI Crawlers Cost You Money? Bandwidth, Server Load, and the Broken Bargain
Every AI crawl is a request your server pays to answer, and training crawlers send few visitors back. The crawl-to-referral numbers for 2025 and 2026, how to measure your own exposure, and when the cost justifies blocking.
The AI Crawler Directory: Main User-Agents, What Each Does, Allow or Block (2026)
Which AI bots should you block? A reference table of the main AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Meta, Amazon, Mistral and more) with each bot's job, robots.txt behaviour, IP-range file and a recommendation.
Anthropic's Crawlers: ClaudeBot, Claude-SearchBot, Claude-User
Anthropic runs ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user-fetch). All three honour robots.txt, including Crawl-delay, which is why a blanket Disallow also stops Claude opening your page for a user.
Should You Block or Allow AI Crawlers? A Decision Rule and the robots.txt to Ship
Decide AI crawler access by bot type: training is your call, search bots cite you, user-fetch bots are visitors. A rule by business type, the robots.txt template, Cloudflare settings and a curl check.
Google's AI Crawlers: Google-Extended, Google-CloudVertexBot, and Gemini
Google-Extended isn't a crawler, it's an opt-out token. How Google-Extended, Google-CloudVertexBot, Google-Agent, Googlebot and the coming AIPREF training signal relate to Gemini and AI features.
How AI Crawlers Work: From Request to Model, Answer, or Visit
How AI crawlers fetch and use your content: the request, user-agent identification, the robots.txt check, why most don't run JavaScript, and the three destinations: model training, a citation index, or a live user.
Meta and Amazon AI Crawlers: Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot
Meta runs Meta-ExternalAgent (training or indexing), Meta-WebIndexer (search), Meta-ExternalFetcher (user-fetch) and FacebookExternalHit (link previews); Amazon splits Amazonbot (may train), Amzn-SearchBot and Amzn-User. How to handle each.
OpenAI's Crawlers: GPTBot, OAI-SearchBot, ChatGPT-User (and OAI-AdsBot)
OpenAI runs separate bots for separate jobs: GPTBot (training), OAI-SearchBot (search), ChatGPT-User (user-fetch) and OAI-AdsBot (ads). What each does, whether to block it, and how to verify it by IP range.
Perplexity's Crawlers: PerplexityBot, Perplexity-User, and the Stealth-Crawling Controversy
PerplexityBot indexes for citations; Perplexity-User fetches pages users ask about and generally ignores robots.txt. Plus the August 2025 Cloudflare report that Perplexity crawled sites that blocked it.
Why are publishers blocking AI crawlers, and should your business do the same?
Who blocks AI bots in 2026 (Reuters Institute, BuzzStream and Elmo data), the licensing deals and their reported amounts, what blocking did to traffic and AI citations, pay-per-crawl, TollBit and RSL, where NYT v. OpenAI and the other lawsuits stand, and a rule by business type with the exact robots.txt and Cloudflare settings.
What You Lose by Blocking AI Search Bots
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Can robots.txt Stop AI Scrapers?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Do AI Crawlers Respect robots.txt?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
llms.txt vs robots.txt: What's the Difference?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Why robots.txt Won't Block AI Bots (and What Actually Does)
Do AI crawlers respect robots.txt? The compliant ones do, user-triggered fetchers often don't, and scrapers ignore it. robots.txt asks; a WAF enforces. Here's the split, bot by bot.
Should I Allow AI Crawlers?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Should I Block GPTBot?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Which AI Bots Should You Block? (And Why robots.txt Won't Stop Them)
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.