Directory Crawlers

Meta-WebIndexer Bot

Builds a search index. Operated by meta.

Identity: Names itself in its user agent only.

How to recognise it

  • User agent contains meta-webindexer

With Double Agent

Sessions from Meta-WebIndexer are labelled bot with agent_id: meta.webindexer, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers ยท Operator docs: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers