Directory Scraping services

Tavily Bot

Collects data. Operated by tavily.

Identity: Names itself in its user agent only.

How to recognise it

  • User agent contains TavilyBot

With Double Agent

Sessions from Tavily are labelled bot with agent_id: tavily.api, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://docs.tavily.com/documentation/search-crawler