Meta and Amazon AI Crawlers: Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot
Meta runs Meta-ExternalAgent (training or indexing), Meta-WebIndexer (search), Meta-ExternalFetcher (user-fetch) and FacebookExternalHit (link previews); Amazon splits Amazonbot (may train), Amzn-SearchBot and Amzn-User. How to handle each.
#ai-crawlers #meta #amazon