What GPTBot does
OpenAI says GPTBot crawls content that may be used in training its generative AI foundation models, and that disallowing it signals a site’s content should not be used for training.
OpenAI treats each of its crawlers separately. Its documentation gives the example of a site that allows OAI-SearchBot to appear in ChatGPT search results while disallowing GPTBot.
When a site allows both GPTBot and OAI-SearchBot, OpenAI says it may use the results of one crawl for both.
| Operator | OpenAI |
|---|---|
| robots.txt token | GPTBot |
| User agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot |
| Used for | OpenAI model training |
| If you block it | Content is not used to train OpenAI models. ChatGPT search is unaffected. |
| Check a visit is real | Match the request’s IP address against the list OpenAI publishes at openai.com/gptbot.json. OpenAI notes that the version number in the user agent may change. |