Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

Meta and Amazon AI Crawlers: Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot

Meta runs Meta-ExternalAgent (training or indexing), Meta-WebIndexer (search), Meta-ExternalFetcher (user-fetch) and FacebookExternalHit (link previews); Amazon splits Amazonbot (may train), Amzn-SearchBot and Amzn-User. How to handle each.

By Andrej Ruckij · · 5 min read

Meta and Amazon AI Crawlers: Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: Meta’s crawler page lists Meta-ExternalAgent (training foundation models, and indexing to improve Meta products), Meta-WebIndexer (Meta AI search), Meta-ExternalFetcher (user-fetch, may bypass robots.txt), Meta-ExternalAds (ads) and FacebookExternalHit (link previews, may bypass robots.txt for integrity checks). Meta publishes no IP ranges there. Amazon documents Amazonbot (may train Amazon AI models), Amzn-SearchBot (search, not training) and Amzn-User (live user fetches, may not follow every robots.txt rule). Amazon doesn’t support Crawl-delay, uses a robots.txt copy up to 30 days old, and reads each subdomain separately.

Part of the AI crawler directory. Meta and Amazon get less attention than OpenAI, Anthropic and Google, but each has a trap: Meta’s training crawler also indexes, and Amazon’s three bot names sit close together.

Meta’s crawlers

User-agentJobrobots.txtRecommendation
Meta-ExternalAgentTraining foundation AI models; also indexing content to improve productsNo bypass statedBlock to opt out of training (may also reduce Meta product indexing)
Meta-WebIndexerImprove Meta AI search resultsNo bypass statedAllow
Meta-ExternalFetcherFetch individual links at a user’s request, incl. agentic features”May bypass robots.txt rules”Leave alone
Meta-ExternalAdsCrawl for advertising and other business productsCan be disallowedYour call; relevant if you advertise on Meta
FacebookExternalHitCrawl a shared page to build the link previewMay bypass for security or integrity checksLeave alone, or shared links lose previews

Meta-ExternalAgent (meta-externalagent/1.1) isn’t training-only like GPTBot or ClaudeBot. Meta says it crawls “for use cases such as training foundation AI models” and also indexes content to improve products, so a block is a training opt-out with a possible side effect on Meta products. Meta AI search has its own token, Meta-WebIndexer (meta-webindexer/1.1), and Meta says allowing it “helps us cite and link to your content.” Meta-ExternalFetcher (meta-externalfetcher/1.1) is the user-fetch agent.

FacebookExternalHit (facebookexternalhit/1.1) fetches your page when someone shares it, to pull the title, description and thumbnail for the link preview; that’s its primary purpose, per Meta. The FacebookBot token some older guides list isn’t on Meta’s current crawler page.

For FacebookExternalHit, Meta’s page calls allow-listing by IP more secure than matching user-agents, but the page lists no IP ranges, so there’s no published Meta file to verify against.

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-WebIndexer
Allow: /

Amazon’s crawlers, mind the naming

Amazon’s crawler page lists three bots, each with its own IP list:

User-agentJobTrainingrobots.txtIP list
Amazonbot (Amazonbot/0.1)Improve Amazon products and services”May be used to train Amazon AI models”Honours user-agent and allow/disallow rulesdeveloper.amazon.com/amazonbot/ip-addresses/
Amzn-SearchBot (Amzn-SearchBot/0.1)Search in Amazon products such as AlexaNot used for generative-AI trainingHonours it; if not named, follows the rules you give other search botsdeveloper.amazon.com/amazonbot/searchbot-ip-addresses/
Amzn-User (Amzn-User/0.1)User-initiated requests, such as live data for AlexaNot used for training”May not follow all robots.txt directives”developer.amazon.com/amazonbot/live-ip-addresses/

Don’t block on the “Amazon” or “Amzn” substring. If you want Alexa-style answers but not training, the distinction is the whole point.

Amazon’s rules also differ from other vendors in ways that bite:

  • No Crawl-delay. Amazon says its crawlers “do not support the crawl-delay directive.” Anthropic does; Amazon doesn’t. To slow Amazonbot you need a firewall rate limit.
  • Up to 30 days of caching. The bots fetch your robots.txt or use a cached copy no older than 30 days, so a new Disallow may not apply right away. If the file can’t be fetched, they treat it as absent. Amazon separately says settings changes may take about 24 hours.
  • Each subdomain separately. shop.example.com needs its own robots.txt; rules on the main domain don’t carry over.
  • Amzn-SearchBot follows search-bot rules when it isn’t named. If you allow other search bots and don’t mention Amzn-SearchBot, it crawls under the same rules.
User-agent: Amazonbot
Disallow: /

User-agent: Amzn-SearchBot
Allow: /
  • Meta: block Meta-ExternalAgent to opt out of training, accepting it may also affect Meta product indexing; allow Meta-WebIndexer if you want to appear in Meta AI search; leave Meta-ExternalFetcher and FacebookExternalHit alone.
  • Amazon: decide on Amazonbot by your training stance; allow Amzn-SearchBot; leave Amzn-User alone. Put the rules on every subdomain and allow for the 30-day cache.

For the full cross-vendor file, use the template in block-or-allow-ai-crawlers. Bots that don’t follow robots.txt need a firewall (robots-txt-vs-waf-ai-bots); verify Amazon traffic by its IP lists, and accept that Meta traffic can only be identified by user-agent unless you build your own IP list.

Key takeaways

  • Meta: Meta-ExternalAgent (training plus product indexing), Meta-WebIndexer (search), Meta-ExternalFetcher (user-fetch, may bypass robots.txt), FacebookExternalHit (link previews, may bypass for integrity checks). No published IP ranges.
  • Amazon: Amazonbot (may train), Amzn-SearchBot (search, no training), Amzn-User (user fetches, may not follow every rule). Each has an IP list.
  • Amazon ignores Crawl-delay, caches robots.txt for up to 30 days and treats each subdomain separately.
  • An unnamed Amzn-SearchBot follows whatever you tell other search bots.

Sources

#ai-crawlers#meta#amazon#robots-txt#bot-management