Directory Crawlers

ExaSearchBot Bot

Builds a search index. Operated by exa.

Identity: Signs its requests (Web Bot Auth).

How to recognise it

  • Signed requests: Signature-Agent from crawler.exa.ai, keys at https://crawler.exa.ai/.well-known/http-message-signatures-directory
  • User agent contains ExaSearchBot, ExaBot

With Double Agent

Sessions from ExaSearchBot are labelled bot with agent_id: exa.searchbot, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://crawler.exa.ai