#ai-crawlers
34 posts tagged with ai-crawlers.
AI Crawler Access Checklist: 8 Steps for Site Owners
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
AI Crawler Access Control: The Complete Guide (2026)
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
AI Crawler Regulation in the EU and UK (2026): What Site Owners Should Know
The EU AI Act gives machine-readable opt-outs legal weight for AI training, and robots.txt is the one the GPAI Code names; the UK dropped its text-and-data-mining opt-out plan in March 2026 and is waiting. What that means for your robots.txt.
AI Crawler Tarpits and Honeypots: Nepenthes, Anubis, and Cloudflare AI Labyrinth
When blocking isn't enough, some sites trap or slow AI crawlers instead. What Nepenthes, Anubis and Cloudflare AI Labyrinth actually do, and the search-visibility costs of each.
Do AI Crawlers Cost You Money? Bandwidth, Server Load, and the Broken Bargain
Every AI crawl is a request your server pays to answer, and training crawlers send few visitors back. The crawl-to-referral numbers for 2025 and 2026, how to measure your own exposure, and when the cost justifies blocking.
The AI Crawler Directory: Main User-Agents, What Each Does, Allow or Block (2026)
Which AI bots should you block? A reference table of the main AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Meta, Amazon, Mistral and more) with each bot's job, robots.txt behaviour, IP-range file and a recommendation.
Anthropic's Crawlers: ClaudeBot, Claude-SearchBot, Claude-User
Anthropic runs ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user-fetch). All three honour robots.txt, including Crawl-delay, which is why a blanket Disallow also stops Claude opening your page for a user.
Blocking AI Crawlers by IP and ASN (for Stealth Scrapers)
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Should You Block or Allow AI Crawlers? A Decision Rule and the robots.txt to Ship
Decide AI crawler access by bot type: training is your call, search bots cite you, user-fetch bots are visitors. A rule by business type, the robots.txt template, Cloudflare settings and a curl check.
Google's AI Crawlers: Google-Extended, Google-CloudVertexBot, and Gemini
Google-Extended isn't a crawler, it's an opt-out token. How Google-Extended, Google-CloudVertexBot, Google-Agent, Googlebot and the coming AIPREF training signal relate to Gemini and AI features.
How AI Crawlers Work: From Request to Model, Answer, or Visit
How AI crawlers fetch and use your content: the request, user-agent identification, the robots.txt check, why most don't run JavaScript, and the three destinations: model training, a citation index, or a live user.
Meta and Amazon AI Crawlers: Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot
Meta runs Meta-ExternalAgent (training or indexing), Meta-WebIndexer (search), Meta-ExternalFetcher (user-fetch) and FacebookExternalHit (link previews); Amazon splits Amazonbot (may train), Amzn-SearchBot and Amzn-User. How to handle each.
How to Block AI Scrapers: The Complete Enforcement Guide (2026)
robots.txt won't stop scrapers that ignore it. The enforcement layer, with working config: rate limiting, bot verification, WAF rules, IP and ASN blocking, and tarpits as a last resort.
OpenAI's Crawlers: GPTBot, OAI-SearchBot, ChatGPT-User (and OAI-AdsBot)
OpenAI runs separate bots for separate jobs: GPTBot (training), OAI-SearchBot (search), ChatGPT-User (user-fetch) and OAI-AdsBot (ads). What each does, whether to block it, and how to verify it by IP range.
Perplexity's Crawlers: PerplexityBot, Perplexity-User, and the Stealth-Crawling Controversy
PerplexityBot indexes for citations; Perplexity-User fetches pages users ask about and generally ignores robots.txt. Plus the August 2025 Cloudflare report that Perplexity crawled sites that blocked it.
Why are publishers blocking AI crawlers, and should your business do the same?
Who blocks AI bots in 2026 (Reuters Institute, BuzzStream and Elmo data), the licensing deals and their reported amounts, what blocking did to traffic and AI citations, pay-per-crawl, TollBit and RSL, where NYT v. OpenAI and the other lawsuits stand, and a rule by business type with the exact robots.txt and Cloudflare settings.
How to Verify a Real AI Bot (IP Ranges, Reverse DNS)
User-agent strings are spoofable, so verify AI bots by their published IP-range files, reverse DNS or request signatures, not by name. How to confirm a request really is GPTBot, ClaudeBot or PerplexityBot.
What You Lose by Blocking AI Search Bots
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Can robots.txt Stop AI Scrapers?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Do AI Crawlers Respect robots.txt?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Does Blocking AI Bots Hurt Your SEO or AI Visibility?
Blocking AI training bots in robots.txt doesn't hurt Google SEO, and Google-Extended doesn't touch AI Overviews. Blocking AI search bots, or a plain Cloudflare block that reaches Googlebot, does cost visibility.
Does Cloudflare Block AI Crawlers by Default?
For new domains, partly. Since Sep 15, 2026 new ad-supported domains refuse AI training and block agent bots on ad pages; search stays allowed. Existing domains keep their settings. Edge rules override robots.txt.
Does llms.txt Actually Work? What the Server Logs Show in 2026
Which AI systems actually fetch llms.txt (Ahrefs, EZY and SE Ranking data), what Google, OpenAI, Anthropic and Perplexity say on their own pages, where the file measurably helps (docs read by coding agents), how to check your own logs, what it costs, how it fails, and a decision rule by site type.
GPTBot vs OAI-SearchBot: What's the Difference?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
How to Create an llms.txt File (with Template)
A step-by-step guide to writing an llms.txt file: the markdown format, what to include, where to put it, and the optional llms-full.txt, plus a note on what it does and doesn't do.
Is llms.txt Worth It?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
llms.txt Best Practices: Format, Curation, and Maintenance
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
llms.txt: The Complete Guide (2026)
What llms.txt is, how to create one, how it differs from robots.txt and sitemap.xml, where it measurably helps (docs read by coding agents) and where it doesn't (AI search), and a rule for deciding by site type.
llms.txt vs robots.txt: What's the Difference?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
llms.txt vs sitemap.xml: What's the Difference?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Why robots.txt Won't Block AI Bots (and What Actually Does)
Do AI crawlers respect robots.txt? The compliant ones do, user-triggered fetchers often don't, and scrapers ignore it. robots.txt asks; a WAF enforces. Here's the split, bot by bot.
Should I Allow AI Crawlers?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Should I Block GPTBot?
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.
Which AI Bots Should You Block? (And Why robots.txt Won't Stop Them)
Replaced in November 2026: this answer now sits in a longer, current page on the same question. Linked here.