Directory Crawlers

AdsBot-Google Bot

Checks ads. Operated by google.

Identity: Publishes its IP addresses.

How to recognise it

  • Published IP list: https://developers.google.com/static/crawling/ipranges/special-crawlers.json
  • Reverse DNS ends in googlebot.com, google.com, googleusercontent.com (then forward-confirm)
  • User agent contains Mediapartners-Google, APIs-Google, Google-Safety

With Double Agent

Sessions from AdsBot-Google are labelled bot with agent_id: google.adsbot, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers ยท Operator docs: https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers