Directory Scraping services

Scrapy / Zyte Bot

Collects data. Operated by zyte.

Identity: Names itself in its user agent only.

How to recognise it

  • User agent contains Scrapy

With Double Agent

Sessions from Scrapy / Zyte are labelled bot with agent_id: zyte.scrapy, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://github.com/scrapy/scrapy/blob/master/scrapy/settings/default_settings.py