Directory Crawlers

Meta-ExternalAgent Bot

Trains AI models. Operated by meta.

Identity: Signs its requests (Web Bot Auth).

How to recognise it

  • Signed requests: Signature-Agent from www.meta.com, keys at https://www.meta.com/.well-known/http-message-signatures-directory
  • User agent contains meta-externalagent, FacebookBot

Allow or block

To opt out, add to robots.txt:

User-agent: meta-externalagent
Disallow: /

With Double Agent

Sessions from Meta-ExternalAgent are labelled bot with agent_id: meta.externalagent, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers ยท Operator docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers