Get started

Directory

Who’s at the door.

226 AI agents, fetchers, crawlers and scrapers, and how to tell each one apart.

agents.jsonView as Markdown

Agent browsers

An AI drives the visitor’s own browser.

NameOperatorDoesIdentity
ChatGPT Atlas AgentopenaiActs for a personDoesn’t identify itself
Claude in Chrome AgentanthropicActs for a personDoesn’t identify itself
Comet AgentperplexityActs for a personDoesn’t identify itself
Copilot Actions in Edge AgentmicrosoftActs for a personDoesn’t identify itself
Dia AgentbrowsercompanyActs for a personDoesn’t identify itself
Fellou AgentfellouActs for a personDoesn’t identify itself
Gemini in Chrome AgentgoogleActs for a personDoesn’t identify itself
Genspark Browser AgentgensparkActs for a personDoesn’t identify itself
Opera Neon AgentoperaActs for a personDoesn’t identify itself

Cloud agents

An AI drives a browser in someone’s cloud.

NameOperatorDoesIdentity
AGI Agent AgentagiActs for a personSigns its requests (Web Bot Auth)
Anchor Browser AgentanchorActs for a personSigns its requests (Web Bot Auth)
Bedrock AgentCore Browser AgentamazonActs for a personSigns its requests (Web Bot Auth)
ChatGPT agent AgentopenaiActs for a personSigns its requests (Web Bot Auth)
Cloudflare Browser Run AgentcloudflareActs for a personSigns its requests (Web Bot Auth)
Firmly AgentfirmlyBuys or pays for a personSigns its requests (Web Bot Auth)
Google-Agent AgentgoogleActs for a personSigns its requests (Web Bot Auth)
Henry AgenthenryBuys or pays for a personSigns its requests (Web Bot Auth)
Hyperbrowser AgenthyperbrowserActs for a personDoesn’t identify itself
Kernel AgentkernelActs for a personSigns its requests (Web Bot Auth)
Manus AgentmanusActs for a personSigns its requests (Web Bot Auth)
Nekuda AgentnekudaBuys or pays for a personSigns its requests (Web Bot Auth)
Rye AgentryeBuys or pays for a personSigns its requests (Web Bot Auth)
Steel AgentsteelActs for a personDoesn’t identify itself
Stripe Link AgentstripeBuys or pays for a personSigns its requests (Web Bot Auth)
Strivve AgentstrivveBuys or pays for a personSigns its requests (Web Bot Auth)
Twin AgenttwinBuys or pays for a personSigns its requests (Web Bot Auth)

Agent frameworks

Libraries developers build agents with.

NameOperatorDoesIdentity
Amazon Nova Act AgentamazonActs for a personNames itself in its user agent only
Browser Use Agentbrowser-useActs for a personDoesn’t identify itself
Browserbase / Stagehand AgentbrowserbaseActs for a personSigns its requests (Web Bot Auth)
Goose AgentblockActs for a personSigns its requests (Web Bot Auth)
Skyvern AgentskyvernActs for a personDoesn’t identify itself

Assistants fetching for a user

Fetched because a person asked an assistant.

NameOperatorDoesIdentity
Amzn-User AgentamazonActs for a personNames itself in its user agent only
BuddyBot AgentbuddybotActs for a personNames itself in its user agent only
ChatGPT connectors AgentopenaiActs for a personPublishes its IP addresses
ChatGPT-User AgentopenaiActs for a personPublishes its IP addresses
Claude Code AgentanthropicActs for a personPublishes its IP addresses
Claude-User AgentanthropicActs for a personPublishes its IP addresses
Cursor AgentcursorActs for a personSigns its requests (Web Bot Auth)
Devin AgentcognitionActs for a personNames itself in its user agent only
GeistHaus PageFetcher AgentgeisthausActs for a personNames itself in its user agent only
Gemini CLI AgentgoogleActs for a personNames itself in its user agent only
Gemini Deep Research AgentgoogleActs for a personNames itself in its user agent only
Google-GeminiNotebook AgentgoogleActs for a personPublishes its IP addresses
Kagi fetcher AgentkagiActs for a personNames itself in its user agent only
Kimi-User AgentmoonshotActs for a personNames itself in its user agent only
LinerBot AgentlinerActs for a personNames itself in its user agent only
Meta-ExternalFetcher AgentmetaActs for a personSigns its requests (Web Bot Auth)
MistralAI-User AgentmistralActs for a personPublishes its IP addresses
Mozilla Tabstack AgentmozillaActs for a personNames itself in its user agent only
opencode AgentsstActs for a personNames itself in its user agent only
Perplexity-User AgentperplexityActs for a personPublishes its IP addresses
PhindBot AgentphindActs for a personNames itself in its user agent only
Poggio-Citations AgentpoggioActs for a personNames itself in its user agent only
QualifiedBot AgentqualifiedActs for a personNames itself in its user agent only
UseAI AgentuseaiActs for a personNames itself in its user agent only
wpbot AgentquantumcloudActs for a personNames itself in its user agent only
WRTNBot AgentwrtnActs for a personNames itself in its user agent only
Discordbot BotdiscordLink previewsNames itself in its user agent only
Embedly BotembedlyLink previewsNames itself in its user agent only
facebookexternalhit BotmetaLink previewsNames itself in its user agent only
Google user-triggered fetchers BotgoogleFetches feeds and filesPublishes its IP addresses
Iframely BotiframelyLink previewsNames itself in its user agent only
LinkedInBot BotlinkedinLink previewsNames itself in its user agent only
Pinterestbot BotpinterestLink previewsVerifiable by reverse DNS
redditbot BotredditLink previewsNames itself in its user agent only
Slackbot BotslackLink previewsNames itself in its user agent only
Snap URL Preview BotsnapLink previewsNames itself in its user agent only
TelegramBot BottelegramLink previewsNames itself in its user agent only
Twitterbot BotxLink previewsNames itself in its user agent only
WhatsApp BotmetaLink previewsNames itself in its user agent only

Crawlers

Crawl the web for training, search or data.

NameOperatorDoesIdentity
360Spider BotqihooBuilds a search indexNames itself in its user agent only
AddSearchBot BotaddsearchBuilds a search indexNames itself in its user agent only
AdsBot-Google BotgoogleChecks adsPublishes its IP addresses
AgentTimes BotagenttimesCollects dataNames itself in its user agent only
AhrefsBot BotahrefsSEO researchPublishes its IP addresses
AI2Bot BotallenaiTrains AI modelsNames itself in its user agent only
aiHitBot BotaihitCollects dataNames itself in its user agent only
AIWebIndex BotlyrenthBuilds a search indexNames itself in its user agent only
Amazon AI services (Kendra, Q Business, Bedrock) BotamazonCollects dataNames itself in its user agent only
Amazonbot BotamazonTrains AI modelsNames itself in its user agent only
Amzn-SearchBot BotamazonBuilds a search indexNames itself in its user agent only
Andibot BotandiBuilds a search indexNames itself in its user agent only
Anomura BotdireqtBuilds a search indexNames itself in its user agent only
Applebot BotappleBuilds a search indexPublishes its IP addresses
Aranet-SearchBot BotaranetBuilds a search indexNames itself in its user agent only
Atlassian BotatlassianCollects dataNames itself in its user agent only
Awario BotawarioCollects dataNames itself in its user agent only
AzureAI-SearchBot BotmicrosoftBuilds a search indexNames itself in its user agent only
Baiduspider BotbaiduBuilds a search indexVerifiable by reverse DNS
Barkrowler BotbabbarSEO researchNames itself in its user agent only
Better Stack BotbetteruptimeMonitors uptime and speedNames itself in its user agent only
bigsur.ai BotbigsurCollects dataNames itself in its user agent only
Bingbot BotmicrosoftBuilds a search indexPublishes its IP addresses
BLEXBot BotwebmeupSEO researchNames itself in its user agent only
Brandwatch BotbrandwatchCollects dataNames itself in its user agent only
Bravebot BotbraveBuilds a search indexNames itself in its user agent only
Brightbot BotbrightdataCollects dataNames itself in its user agent only
BrightEdge BotbrightedgeSEO researchNames itself in its user agent only
Bytespider BotbytedanceTrains AI modelsNames itself in its user agent only
CCBot BotcommoncrawlTrains AI modelsPublishes its IP addresses
Channel3Bot Botchannel3Builds a search indexNames itself in its user agent only
ChatGLM-Spider BotzhipuTrains AI modelsNames itself in its user agent only
Checkly BotchecklyMonitors uptime and speedPublishes its IP addresses
Claude-SearchBot BotanthropicBuilds a search indexPublishes its IP addresses
ClaudeBot BotanthropicTrains AI modelsPublishes its IP addresses
Cloudflare AI Search BotcloudflareBuilds a search indexSigns its requests (Web Bot Auth)
coccocbot BotcoccocBuilds a search indexNames itself in its user agent only
Cohere BotcohereTrains AI modelsNames itself in its user agent only
Cotoyogi BotroisTrains AI modelsNames itself in its user agent only
CragCrawler BotcragsoftwareCollects dataNames itself in its user agent only
Crawlspace BotcrawlspaceCollects dataNames itself in its user agent only
Datadog Synthetics BotdatadogMonitors uptime and speedPublishes its IP addresses
DataForSeoBot BotdataforseoSEO researchNames itself in its user agent only
Datenbank Crawler BotdatenbankCollects dataNames itself in its user agent only
DeepSeekBot BotdeepseekTrains AI modelsNames itself in its user agent only
Diffbot BotdiffbotCollects dataNames itself in its user agent only
DotBot / rogerbot BotmozSEO researchNames itself in its user agent only
DuckAssistBot BotduckduckgoBuilds a search indexSigns its requests (Web Bot Auth)
DuckDuckBot BotduckduckgoBuilds a search indexPublishes its IP addresses
Echobox BotechoboxCollects dataNames itself in its user agent only
ERNIE BotbaiduTrains AI modelsNames itself in its user agent only
ExaSearchBot BotexaBuilds a search indexSigns its requests (Web Bot Auth)
FactSet BotfactsetCollects dataNames itself in its user agent only
FriendlyCrawler BotfriendlycrawlerCollects dataNames itself in its user agent only
Googlebot BotgoogleBuilds a search indexPublishes its IP addresses
GoogleOther BotgoogleTrains AI modelsPublishes its IP addresses
GPTBot BotopenaiTrains AI modelsPublishes its IP addresses
Grok BotxaiTrains AI modelsNames itself in its user agent only
GTmetrix BotgtmetrixMonitors uptime and speedNames itself in its user agent only
HenkBot BothenkbotCollects dataNames itself in its user agent only
iAsk BotiaskBuilds a search indexNames itself in its user agent only
ICC-Crawler BotnictTrains AI modelsNames itself in its user agent only
ImagesiftBot BothiveCollects dataNames itself in its user agent only
imageSpider BotimagespiderCollects dataNames itself in its user agent only
img2dataset BotlaionCollects dataNames itself in its user agent only
ISS Cyber Risk BotissCollects dataNames itself in its user agent only
Kangaroo Bot BotkangarooTrains AI modelsNames itself in its user agent only
KimiBot BotmoonshotTrains AI modelsNames itself in its user agent only
KlaviyoAIBot BotklaviyoCollects dataSigns its requests (Web Bot Auth)
KunatoCrawler BotkunatoCollects dataNames itself in its user agent only
Lighthouse BotgoogleMonitors uptime and speedNames itself in its user agent only
Linguee BotlingueeCollects dataNames itself in its user agent only
LinkupBot BotlinkupBuilds a search indexPublishes its IP addresses
Meltwater BotmeltwaterCollects dataNames itself in its user agent only
Meta-ExternalAgent BotmetaTrains AI modelsSigns its requests (Web Bot Auth)
Meta-WebIndexer BotmetaBuilds a search indexNames itself in its user agent only
MistralAI-Index BotmistralBuilds a search indexPublishes its IP addresses
MistralAI-Training BotmistralTrains AI modelsNames itself in its user agent only
MJ12bot BotmajesticSEO researchNames itself in its user agent only
MojeekBot BotmojeekBuilds a search indexNames itself in its user agent only
MyCentralAIScraperBot BotmycentralaiCollects dataNames itself in its user agent only
NagetBot BotnagetCollects dataNames itself in its user agent only
netEstate BotnetestateCollects dataNames itself in its user agent only
New Relic Synthetics BotnewrelicMonitors uptime and speedNames itself in its user agent only
newsai BotnewsaiCollects dataNames itself in its user agent only
OAI-AdsBot BotopenaiChecks adsPublishes its IP addresses
OAI-SearchBot BotopenaiBuilds a search indexPublishes its IP addresses
Panscient BotpanscientCollects dataNames itself in its user agent only
PerplexityBot BotperplexityBuilds a search indexPublishes its IP addresses
PetalBot BothuaweiBuilds a search indexNames itself in its user agent only
Pingdom BotpingdomMonitors uptime and speedPublishes its IP addresses
Poseidon BotposeidonCollects dataNames itself in its user agent only
Querit BotqueritBuilds a search indexNames itself in its user agent only
QuillBot BotquillbotCollects dataNames itself in its user agent only
Qwantbot BotqwantBuilds a search indexNames itself in its user agent only
Qwen BotalibabaTrains AI modelsNames itself in its user agent only
Reflectionbot BotreflectionCollects dataNames itself in its user agent only
SBIntuitionsBot BotsoftbankTrains AI modelsNames itself in its user agent only
Screaming Frog BotscreamingfrogSEO researchNames itself in its user agent only
SE Ranking BotserankingSEO researchNames itself in its user agent only
SeekportBot BotseekportBuilds a search indexNames itself in its user agent only
SemrushBot BotsemrushSEO researchNames itself in its user agent only
SEOkicks BotseokicksSEO researchNames itself in its user agent only
serpstatbot BotserpstatSEO researchNames itself in its user agent only
SeznamBot BotseznamBuilds a search indexVerifiable by reverse DNS
ShapBot BotparallelBuilds a search indexPublishes its IP addresses
Sidetrade BotsidetradeCollects dataNames itself in its user agent only
Site24x7 Botsite24x7Monitors uptime and speedNames itself in its user agent only
Sitebulb BotsitebulbSEO researchNames itself in its user agent only
Siteimprove BotsiteimproveSEO researchNames itself in its user agent only
Sogou spider BotsogouBuilds a search indexNames itself in its user agent only
StatusCake BotstatuscakeMonitors uptime and speedNames itself in its user agent only
TerraCotta (Ceramic) BotceramicCollects dataNames itself in its user agent only
Thinkbot BotthinkbotCollects dataNames itself in its user agent only
Timpibot BottimpiTrains AI modelsNames itself in its user agent only
UptimeRobot BotuptimerobotMonitors uptime and speedPublishes its IP addresses
Velen BotvelenCollects dataNames itself in its user agent only
WARDBot BotwardbotCollects dataNames itself in its user agent only
WebPageTest BotwebpagetestMonitors uptime and speedNames itself in its user agent only
Webz.io (omgili) BotwebzCollects dataNames itself in its user agent only
Yahoo Slurp BotyahooBuilds a search indexNames itself in its user agent only
YandexAdditional BotyandexTrains AI modelsVerifiable by reverse DNS
YandexBot BotyandexBuilds a search indexVerifiable by reverse DNS
Yeti (Naver) BotnaverBuilds a search indexNames itself in its user agent only
YouBot BotyouBuilds a search indexSigns its requests (Web Bot Auth)
ZanistaBot BotzanistaBuilds a search indexNames itself in its user agent only

Scraping services

Rented scraping, usually disguised.

NameOperatorDoesIdentity
Apify / Crawlee BotapifyCollects dataSigns its requests (Web Bot Auth)
Bright Data BotbrightdataCollects dataDoesn’t identify itself
Crawl4AI Botcrawl4aiCollects dataNames itself in its user agent only
Firecrawl BotfirecrawlCollects dataNames itself in its user agent only
Jina Reader BotjinaCollects dataDoesn’t identify itself
Oxylabs BotoxylabsCollects dataDoesn’t identify itself
ScraperAPI BotscraperapiCollects dataDoesn’t identify itself
Scrapfly BotscrapflyCollects dataDoesn’t identify itself
ScrapingBee BotscrapingbeeCollects dataDoesn’t identify itself
Scrapy / Zyte BotzyteCollects dataNames itself in its user agent only
SerpApi BotserpapiCollects dataDoesn’t identify itself
Tavily BottavilyCollects dataNames itself in its user agent only
ZenRows BotzenrowsCollects dataDoesn’t identify itself

Automation tools

Headless browsers and drivers.

NameOperatorDoesIdentity
Headless Chrome BotchromeCollects dataNames itself in its user agent only
Lightpanda BotlightpandaCollects dataNames itself in its user agent only
PhantomJS / Nightmare BotlegacyCollects dataNames itself in its user agent only
Playwright BotplaywrightCollects dataDoesn’t identify itself
Puppeteer BotpuppeteerCollects dataDoesn’t identify itself
Selenium / WebDriver BotseleniumCollects dataDoesn’t identify itself

HTTP libraries

Scripts with a default user agent.

NameOperatorDoesIdentity
aiohttp BotpythonCollects dataNames itself in its user agent only
Apache-HttpClient BotapacheCollects dataNames itself in its user agent only
axios BotaxiosCollects dataNames itself in its user agent only
curl BotcurlCollects dataNames itself in its user agent only
Go-http-client BotgoCollects dataNames itself in its user agent only
node-fetch BotnodeCollects dataNames itself in its user agent only
okhttp BotsquareCollects dataNames itself in its user agent only
python-httpx BotpythonCollects dataNames itself in its user agent only
python-requests BotpythonCollects dataNames itself in its user agent only
Python-urllib BotpythonCollects dataNames itself in its user agent only
Wget BotgnuCollects dataNames itself in its user agent only