Index My Site

OAI-SearchBot: the crawler behind ChatGPT search

OAI-SearchBot is the crawler OpenAI uses to surface websites in ChatGPT search. If robots.txt or your firewall blocks it, ChatGPT search answers will not show your pages, apart from navigational links.

What OAI-SearchBot does

OpenAI says OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features, and that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

OpenAI’s help page for ChatGPT search says that to make a website eligible, you allow OAI-SearchBot to crawl it and make sure your host or CDN allows traffic from OpenAI’s published search bot IP addresses. Placement is not guaranteed.

OpenAI says a robots.txt change can take about 24 hours to reach ChatGPT search results.

OperatorOpenAI
robots.txt tokenOAI-SearchBot
User agentMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Used forChatGPT search
If you block itThe page is not shown in ChatGPT search answers, apart from navigational links.
Check a visit is realMatch the request’s IP address against openai.com/searchbot.json, and allow those addresses at your firewall or CDN. When it fetches robots.txt, the user agent adds a robots.txt marker.

Source: OpenAI: overview of OpenAI crawlers

Check whether your robots.txt allows OAI-SearchBot

We read the site's robots.txt file and nothing else. Crawl rules say who may visit. They do not show whether a page is indexed or cited.

Results

View robots.txt

How to block or allow OAI-SearchBot in robots.txt

Use the OAI-SearchBot token. Bot protection on your CDN can block it even when robots.txt allows it, so check your firewall rules too.

To block OAI-SearchBot from the whole site:

User-agent: OAI-SearchBot
Disallow: /

To allow it everywhere, even when another group blocks all crawlers:

User-agent: OAI-SearchBot
Allow: /

A crawler named in a group follows only that group, so repeat any shared Disallow lines in its group. See how robots.txt blocks work for the details.

OpenAI's other crawlers

CrawlerUsed forIf you block it
GPTBotOpenAI model trainingContent is not used to train OpenAI models. ChatGPT search is unaffected.
ChatGPT-UserChatGPT, when a person asks it to visit a pageOpenAI says robots.txt rules may not apply to these visits.

Allowed OAI-SearchBot? Get the page indexed

OAI-SearchBot only lets OpenAI crawl. ChatGPT search also sends rewritten searches to partner search providers, and OpenAI’s ChatGPT Enterprise help page names Bing. A page in Bing’s index, and in Google’s, gives ChatGPT more ways to find it.

Submit your pages to send them to Google, and add Bing to send them to Bing as well. It works for any public URL, including reviews and posts on other sites that mention you. See what each AI answer needs before it can cite a page.

Read next: How to get cited by ChatGPT.

Frequently asked questions

Allow OAI-SearchBot in robots.txt and allow OpenAI’s published search bot IP addresses at your host or CDN. OpenAI has no submission form, and it says placement is not guaranteed.

Source: OpenAI: ChatGPT search

OpenAI says it can take about 24 hours from a robots.txt update for ChatGPT search results to adjust.

Yes. OpenAI says each setting is independent, and gives allowing OAI-SearchBot while disallowing GPTBot as an example.

No. ChatGPT-User visits a page when a person asks ChatGPT to, and OpenAI says robots.txt rules may not apply to those visits. It is not used to decide what appears in ChatGPT search.