Identity: Names itself in its user agent only.
How to recognise it
- User agent contains
FirecrawlAgent
Allow or block
To opt out, add to robots.txt:
User-agent: FirecrawlAgent
Disallow: /With Double Agent
Sessions from Firecrawl are labelled bot with agent_id: firecrawl.api, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md
Source: https://github.com/mendableai/firecrawl