---
title: "GPTBot: OpenAI's training crawler and how to block it"
description: "GPTBot crawls content that may be used to train OpenAI models. What blocking it does, why it does not affect ChatGPT search, and the robots.txt rules to use."
canonical: "https://indexmysite.app/tools/ai-crawler-checker/gptbot"
language: "en"
---

# GPTBot: OpenAI's training crawler and how to block it

GPTBot is the crawler OpenAI uses to collect content that may train its models. Blocking it keeps your pages out of that training and leaves ChatGPT search alone, because ChatGPT search uses a different crawler, OAI-SearchBot.

## What GPTBot does

OpenAI says GPTBot crawls content that may be used in training its generative AI foundation models, and that disallowing it signals a site’s content should not be used for training.

OpenAI treats each of its crawlers separately. Its documentation gives the example of a site that allows OAI-SearchBot to appear in ChatGPT search results while disallowing GPTBot.

When a site allows both GPTBot and OAI-SearchBot, OpenAI says it may use the results of one crawl for both.

<table><tbody><tr><th scope="row">Operator</th><td>OpenAI</td></tr><tr><th scope="row">robots.txt token</th><td><code>GPTBot</code></td></tr><tr><th scope="row">User agent</th><td><code class="block break-all">Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot</code></td></tr><tr><th scope="row">Used for</th><td>OpenAI model training</td></tr><tr><th scope="row">If you block it</th><td>Content is not used to train OpenAI models. ChatGPT search is unaffected.</td></tr><tr><th scope="row">Check a visit is real</th><td>Match the request’s IP address against the list OpenAI publishes at openai.com/gptbot.json. OpenAI notes that the version number in the user agent may change.</td></tr></tbody></table>

Source: [OpenAI: overview of OpenAI crawlers](https://platform.openai.com/docs/bots)

## Check whether your robots.txt allows GPTBot

Use the [AI crawler checker](https://indexmysite.app/tools/ai-crawler-checker) to read a site's robots.txt and see whether GPTBot and the other search, AI and training crawlers may visit a page.

## How to block or allow GPTBot in robots.txt

Use the GPTBot token. A GPTBot group affects training only, so it is safe to block GPTBot while allowing OAI-SearchBot.

To block GPTBot from the whole site:

```
User-agent: GPTBot
Disallow: /
```

To allow it everywhere, even when another group blocks all crawlers:

```
User-agent: GPTBot
Allow: /
```

A crawler named in a group follows only that group, so repeat any shared Disallow lines in its group. See [how robots.txt blocks work](https://indexmysite.app/blocked-by-robots-txt) for the details.

## OpenAI's other crawlers

| Crawler | Used for | If you block it |
| --- | --- | --- |
| [`OAI-SearchBot`](https://indexmysite.app/tools/ai-crawler-checker/oai-searchbot) | ChatGPT search | The page is not shown in ChatGPT search answers, apart from navigational links. |
| `ChatGPT-User` | ChatGPT, when a person asks it to visit a page | OpenAI says robots.txt rules may not apply to these visits. |

## Blocking GPTBot keeps you in search

OpenAI says each crawler setting is independent, so a GPTBot block does not remove your pages from ChatGPT search. To appear there, keep OAI-SearchBot allowed in robots.txt and at your firewall. Google AI Overviews follow your Googlebot rules, and Perplexity follows PerplexityBot’s.

Read next: [Where ChatGPT gets its information](https://indexmysite.app/where-does-chatgpt-get-its-information).

## Frequently asked questions

### If I block GPTBot, will ChatGPT stop showing my site?

No. GPTBot is for model training. ChatGPT search uses OAI-SearchBot, and OpenAI says a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot.

Source: [OpenAI: overview of OpenAI crawlers](https://platform.openai.com/docs/bots)

### Should I block GPTBot?

Block it if you do not want your content used to train OpenAI models. It has no effect on ChatGPT search, Google or Bing, so the choice only concerns training.

### What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content that may train OpenAI models. OAI-SearchBot surfaces websites in ChatGPT search results. Each follows its own robots.txt group.

### Does blocking GPTBot remove my content from existing models?

OpenAI describes the block as a signal that the site’s content should not be used in training. It does not say content collected earlier is removed.
