Directory Crawlers

ClaudeBot Bot

Trains AI models. Operated by anthropic.

Identity: Publishes its IP addresses.

How to recognise it

  • Published IP list: https://claude.com/crawling/bots.json
  • User agent contains ClaudeBot, Claude-Web, anthropic-ai

Allow or block

To opt out, add to robots.txt:

User-agent: ClaudeBot
Disallow: /

With Double Agent

Sessions from ClaudeBot are labelled bot with agent_id: anthropic.claudebot, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://support.claude.com/en/articles/8896518 ยท Operator docs: https://support.claude.com/en/articles/8896518