Index My Site

GPTBot: OpenAI's training crawler and how to block it

GPTBot is the crawler OpenAI uses to collect content that may train its models. Blocking it keeps your pages out of that training and leaves ChatGPT search alone, because ChatGPT search uses a different crawler, OAI-SearchBot.

What GPTBot does

OpenAI says GPTBot crawls content that may be used in training its generative AI foundation models, and that disallowing it signals a site’s content should not be used for training.

OpenAI treats each of its crawlers separately. Its documentation gives the example of a site that allows OAI-SearchBot to appear in ChatGPT search results while disallowing GPTBot.

When a site allows both GPTBot and OAI-SearchBot, OpenAI says it may use the results of one crawl for both.

OperatorOpenAI
robots.txt tokenGPTBot
User agentMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Used forOpenAI model training
If you block itContent is not used to train OpenAI models. ChatGPT search is unaffected.
Check a visit is realMatch the request’s IP address against the list OpenAI publishes at openai.com/gptbot.json. OpenAI notes that the version number in the user agent may change.

Source: OpenAI: overview of OpenAI crawlers

Check whether your robots.txt allows GPTBot

We read the site's robots.txt file and nothing else. Crawl rules say who may visit. They do not show whether a page is indexed or cited.

Results

View robots.txt

How to block or allow GPTBot in robots.txt

Use the GPTBot token. A GPTBot group affects training only, so it is safe to block GPTBot while allowing OAI-SearchBot.

To block GPTBot from the whole site:

User-agent: GPTBot
Disallow: /

To allow it everywhere, even when another group blocks all crawlers:

User-agent: GPTBot
Allow: /

A crawler named in a group follows only that group, so repeat any shared Disallow lines in its group. See how robots.txt blocks work for the details.

OpenAI's other crawlers

CrawlerUsed forIf you block it
OAI-SearchBotChatGPT searchThe page is not shown in ChatGPT search answers, apart from navigational links.
ChatGPT-UserChatGPT, when a person asks it to visit a pageOpenAI says robots.txt rules may not apply to these visits.

Blocking GPTBot keeps you in search

OpenAI says each crawler setting is independent, so a GPTBot block does not remove your pages from ChatGPT search. To appear there, keep OAI-SearchBot allowed in robots.txt and at your firewall. Google AI Overviews follow your Googlebot rules, and Perplexity follows PerplexityBot’s.

Read next: Where ChatGPT gets its information.

Frequently asked questions

No. GPTBot is for model training. ChatGPT search uses OAI-SearchBot, and OpenAI says a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot.

Source: OpenAI: overview of OpenAI crawlers

Block it if you do not want your content used to train OpenAI models. It has no effect on ChatGPT search, Google or Bing, so the choice only concerns training.

GPTBot collects content that may train OpenAI models. OAI-SearchBot surfaces websites in ChatGPT search results. Each follows its own robots.txt group.

OpenAI describes the block as a signal that the site’s content should not be used in training. It does not say content collected earlier is removed.