Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

AI Crawler Tarpits and Honeypots: Nepenthes, Anubis, and Cloudflare AI Labyrinth

When blocking isn't enough, some sites trap or slow AI crawlers instead. What Nepenthes, Anubis and Cloudflare AI Labyrinth actually do, and the search-visibility costs of each.

By Andrej Ruckij · · 5 min read

AI Crawler Tarpits and Honeypots: Nepenthes, Anubis, and Cloudflare AI Labyrinth

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: Tarpits fight crawlers that ignore robots.txt by wasting their time instead of just blocking them. Nepenthes traps any crawler in an endless maze of generated pages, search engines included, so its author warns it will likely get you dropped from Google. Anubis is a proof-of-work gate rather than a tarpit, and its default policy blocks AI search bots too. Cloudflare AI Labyrinth is an opt-in decoy maze whose links are nofollow; Cloudflare says it has no SEO effect. Most sites don’t need any of them.

This is the last rung of the enforcement ladder, for crawlers that ignore robots.txt and get past rate limits and firewall rules.

Why tarpits exist

A plain block returns a 403 and the crawler moves on. A tarpit accepts the request and feeds the crawler something endless or expensive, burning its time, bandwidth and compute. The aim is to make crawling you costly for the scrapers a Disallow never stops (robots-txt-vs-waf-ai-bots). A honeypot is the detection cousin: a hidden link no human would follow, so anything that requests it is a bot and can be banned.

Nepenthes, the infinite maze

Nepenthes, named after the carnivorous pitcher plant, is an open-source tarpit that serves an endless series of randomly generated pages linking only to each other. A crawler that wanders in keeps following links and wasting resources. It’s often paired with fail2ban to ban the IPs that fell in.

The catch is that it doesn’t tell AI crawlers from search crawlers. Its author told heise it will “eat just about anything that finds it’s way inside” and that anyone who puts it on their site will most likely be kicked out of Google search. Treat it as an option only for sites that don’t care about search traffic.

A related tool, iocaine, takes the same approach of feeding unwanted bots junk, with the stated aim of poisoning them so they go away.

Anubis, a proof-of-work gate

Anubis works differently. It puts a gate in front of your site: each visitor’s browser has to solve a small SHA-256 proof-of-work puzzle before it gets through. That’s cheap for one person and expensive at crawl scale.

Three things to know before you deploy it:

  • It needs JavaScript by default. The challenge runs in the visitor’s browser. A no-JavaScript “meta refresh” challenge exists, but it is off by default while its false-positive rate is tested.
  • Its default policy blocks AI search. The shipped config imports ai-block-aggressive, which blocks every AI-associated user agent regardless of purpose, including AI search crawlers and user-fetch bots. If you want to keep appearing in ChatGPT or Perplexity answers, switch to ai-block-moderate, which allows OAI-SearchBot, PerplexityBot, ChatGPT-User and Perplexity-User. It ships commented out.
  • It can catch good bots. The project itself calls Anubis “a bit of a nuclear response” that can get in the way of bots like the Internet Archive unless you allowlist them.

Cloudflare AI Labyrinth

AI Labyrinth (announced March 2025) is Cloudflare’s managed decoy. It’s opt-in: you turn it on under Security Settings > Bot traffic. Once on, it adds invisible links to your pages that only bots see, and when Cloudflare detects inappropriate bot activity, those links lead the crawler into a maze of realistic but irrelevant AI-generated pages. The links carry nofollow, and Cloudflare states they don’t affect your SEO or how your site looks. Of the three tools here, it’s the only one that doesn’t put your search visibility at stake, as far as its vendor says. Cloudflare’s wider defaults are in does-cloudflare-block-ai-crawlers.

The tradeoffs

  • Search visibility. Nepenthes can get you de-indexed; Anubis on its default policy blocks AI search bots. Check this before anything else.
  • Your own resources. A self-hosted maze serves real requests and costs bandwidth and compute.
  • Collateral damage. Challenges can stop legitimate clients, from no-JavaScript visitors to archive bots.
  • Arms race. Crawlers adapt; none of these is a permanent fix.
  • Most sites don’t need them. robots.txt, a firewall and rate limiting (how-to-block-ai-scrapers) cover the majority. Tarpits make sense under genuine, persistent scraper pressure.

Key takeaways

  • Tarpits waste a crawler’s resources; honeypots detect bots with hidden bait.
  • Nepenthes doesn’t distinguish search crawlers; its author warns it will likely get you kicked out of Google.
  • Anubis is a proof-of-work gate that needs JavaScript by default; switch it to ai-block-moderate if you want AI search to reach you.
  • Cloudflare AI Labyrinth is opt-in, uses nofollow decoy links, and Cloudflare says it has no SEO effect.
  • Start with robots.txt, a firewall and rate limits; reserve tarpits for sustained scraping.

Sources

#ai-crawlers#tarpit#honeypot#anubis#nepenthes#waf