Directory Crawlers

YandexBot Bot

Builds a search index. Operated by yandex.

Identity: Verifiable by reverse DNS.

How to recognise it

  • Reverse DNS ends in yandex.ru, yandex.net, yandex.com (then forward-confirm)
  • User agent contains YandexBot, YandexImages, YandexMobileBot

With Double Agent

Sessions from YandexBot are labelled bot with agent_id: yandex.yandexbot, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots