Directory Crawlers

Bingbot Bot

Builds a search index. Operated by microsoft.

Identity: Publishes its IP addresses.

How to recognise it

  • Published IP list: https://www.bing.com/toolbox/bingbot.json
  • Reverse DNS ends in search.msn.com (then forward-confirm)
  • User agent contains bingbot, adidxbot, BingPreview, MicrosoftPreview

Allow or block

To opt out, add to robots.txt:

User-agent: bingbot
Disallow: /

With Double Agent

Sessions from Bingbot are labelled bot with agent_id: microsoft.bingbot, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0