Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

Perplexity's Crawlers: PerplexityBot, Perplexity-User, and the Stealth-Crawling Controversy

PerplexityBot indexes for citations; Perplexity-User fetches pages users ask about and generally ignores robots.txt. Plus the August 2025 Cloudflare report that Perplexity crawled sites that blocked it.

By Andrej Ruckij · · 4 min read

Perplexity’s Crawlers: PerplexityBot, Perplexity-User, and the Stealth-Crawling Controversy

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: PerplexityBot indexes pages so Perplexity can cite and link them in its answers; allow it. Perplexity-User fetches a page when a user asks, and Perplexity’s own docs say it “generally ignores robots.txt rules.” Both have IP-range files on perplexity.ai. The caution: in August 2025 Cloudflare reported Perplexity crawling sites that had blocked it, using a generic browser identity. If you really need Perplexity out, robots.txt isn’t enough.

Part of the AI crawler directory. Perplexity is the major vendor where “does it comply?” has the most complicated answer.

The two Perplexity bots

User-agentJobrobots.txtIP rangesRecommendation
PerplexityBotSurface and link sites in Perplexity resultsFollows itwww.perplexity.ai/perplexitybot.jsonAllow
Perplexity-UserFetch a page a user asked aboutGenerally ignores itwww.perplexity.ai/perplexity-user.jsonLeave alone

Note the domain: the IP files sit on perplexity.ai, not perplexity.com.

PerplexityBot, allow it

PerplexityBot (PerplexityBot/1.0) is “designed to surface and link websites in search results on Perplexity.” Perplexity says it doesn’t crawl content for AI foundation models, and recommends allowing it. Changes to your rules may take up to 24 hours to show up in Perplexity’s systems.

Perplexity-User ignores robots.txt on purpose

Perplexity-User (Perplexity-User/1.0) visits a page when a user’s question needs it, and Perplexity says it isn’t used for crawling or for collecting training data. Because a person asked for the fetch, Perplexity says it “generally ignores robots.txt rules.” You’d rarely want to block a visitor anyway, but it means a Disallow won’t stop it.

The August 2025 stealth-crawling report

In August 2025 Cloudflare reported Perplexity crawling sites that had explicitly disallowed it, rotating user-agents and source networks and using a generic “Chrome on macOS” identity to get past blocks on its declared crawler. Cloudflare removed Perplexity from its verified-bots list and added detection rules. Perplexity disputed the findings. Cloudflare’s verified-bot programme now requires Web Bot Auth signatures or IP validation, and Perplexity’s crawler documentation doesn’t mention Web Bot Auth.

The lesson holds whatever the merits of the dispute: a named bot’s compliance can’t be assumed. To keep Perplexity, or anything else, out for real, enforce at the firewall and verify by IP range. See robots-txt-vs-waf-ai-bots for why robots.txt only asks, and how-to-block-ai-scrapers for the firewall side.

Perplexity’s robots.txt line

User-agent: PerplexityBot
Allow: /

There’s no point adding a Perplexity-User line, since it generally ignores robots.txt. If you have a reason to exclude Perplexity entirely, do it at the firewall. For the full cross-vendor file, use the template in block-or-allow-ai-crawlers.

Key takeaways

  • PerplexityBot cites and links you; allow it. Perplexity says it isn’t used for training.
  • Perplexity-User is a visitor and generally ignores robots.txt.
  • Both IP files are on perplexity.ai.
  • August 2025: Cloudflare reported Perplexity crawling sites that blocked it and de-listed it as a verified bot. Compliance can’t be assumed; enforcement is a firewall job.

Sources

#ai-crawlers#perplexity#robots-txt#waf#bot-management