Directory Crawlers

OAI-SearchBot Bot

Builds a search index. Operated by openai.

Identity: Publishes its IP addresses.

How to recognise it

  • Published IP list: https://openai.com/searchbot.json
  • User agent contains OAI-SearchBot

Allow or block

To opt out, add to robots.txt:

User-agent: OAI-SearchBot
Disallow: /

With Double Agent

Sessions from OAI-SearchBot are labelled bot with agent_id: openai.oai-searchbot, and tagged in your analytics. Nothing is blocked unless you choose to. Install: https://doubleagent.so/install.md

Source: https://developers.openai.com/api/docs/bots ยท Operator docs: https://developers.openai.com/api/docs/bots