Directory Scraping services

Apify / Crawlee Bot

Collects data. Operated by apify.

Identity: Signs its requests (Web Bot Auth).

How to recognise it

Signed requests: Signature-Agent from api.apify.com, keys at https://api.apify.com/.well-known/http-message-signatures-directory
User agent contains ApifyBot, ApifyWebsiteContentCrawler

With Double Agent

Sessions from Apify / Crawlee are labelled bot with agent_id: apify.crawlee, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://docs.apify.com