Directory Crawlers

Applebot Bot

Builds a search index. Operated by apple.

Identity: Publishes its IP addresses.

How to recognise it

  • Published IP list: https://search.developer.apple.com/applebot.json
  • Reverse DNS ends in applebot.apple.com (then forward-confirm)
  • User agent contains Applebot

Allow or block

To opt out, add to robots.txt:

User-agent: Applebot-Extended
Disallow: /

With Double Agent

Sessions from Applebot are labelled bot with agent_id: apple.applebot, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://support.apple.com/en-us/119829