Directory Crawlers

AhrefsBot Bot

SEO research. Operated by ahrefs.

Identity: Publishes its IP addresses.

How to recognise it

  • Published IP list: https://api.ahrefs.com/v3/public/crawler-ip-ranges
  • Reverse DNS ends in ahrefs.com, ahrefs.net (then forward-confirm)
  • User agent contains AhrefsBot, AhrefsSiteAudit

With Double Agent

Sessions from AhrefsBot are labelled bot with agent_id: ahrefs.ahrefsbot, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://ahrefs.com/robot