Identity: Publishes its IP addresses.
How to recognise it
- Published IP list: https://index.commoncrawl.org/ccbot.json
- Reverse DNS ends in
crawl.commoncrawl.org(then forward-confirm) - User agent contains
CCBot
Allow or block
To opt out, add to robots.txt:
User-agent: CCBot
Disallow: /With Double Agent
Sessions from CCBot are labelled bot with agent_id: commoncrawl.ccbot, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md
Source: https://commoncrawl.org/ccbot