AI answers start with crawling and indexing
An AI answer can only link to a page its search system has crawled. Google says a page must be indexed and eligible for a snippet to appear as a link in AI Overviews or AI Mode, and the site must not be opted out in Search Console. ChatGPT, Perplexity and Claude run their own search crawlers. If robots.txt blocks one of them, that product cannot crawl your page for its answers, however good the page is.
Training crawlers are different. Blocking GPTBot, ClaudeBot or Google-Extended keeps your content out of model training. Your pages stay in search and AI answers, so many sites block those and allow the rest. See what each AI answer needs before it can cite a page.
The crawlers this tool checks
Search and AI answers
| Crawler | Used for | If you block it |
|---|---|---|
Googlebot | Google Search, AI Overviews and AI Mode | Google cannot crawl the page for Search, including AI Overviews and AI Mode. Source |
Bingbot | Bing and Copilot | Bing cannot crawl the page for its index. Source |
OAI-SearchBot | ChatGPT search | The page is not shown in ChatGPT search answers, apart from navigational links. Source |
Claude-SearchBot | Claude search | Claude does not index the page for search, which may reduce its visibility in answers. Source |
PerplexityBot | Perplexity search | Perplexity does not surface and link the page in its search results. Source |
Applebot | Siri, Spotlight and Safari | Apple cannot crawl the page for search across its devices. Source |
Visits a person asks for
| Crawler | Used for | If you block it |
|---|---|---|
ChatGPT-User | ChatGPT, when a person asks it to visit a page | OpenAI says robots.txt rules may not apply to these visits. Source |
Claude-User | Claude, when a person asks it to visit a page | Claude cannot fetch the page to answer a question. Source |
Perplexity-User | Perplexity, when a person asks it to visit a page | Perplexity says these visits generally ignore robots.txt. Source |
Model training only
| Crawler | Used for | If you block it |
|---|---|---|
GPTBot | OpenAI model training | Content is not used to train OpenAI models. ChatGPT search is unaffected. Source |
ClaudeBot | Anthropic model training | Future content is left out of Anthropic training data. Claude search is unaffected. Source |
Google-Extended | Gemini training and grounding | Gemini apps do not train on or ground answers with the page. Google Search, including AI Overviews, is unaffected. Source |
Applebot follows your Googlebot rules when no group names Applebot, so a Googlebot block also blocks Apple's search.
A robots.txt that keeps you in AI answers
This blocks the three training crawlers and allows everything else. A crawler named in a group follows only that group, so if you name Googlebot or another search crawler anywhere in the file, repeat any shared Disallow lines in its group.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /Allowed everywhere? Get the page indexed
Crawl rules let crawlers in. Submitting a page brings them to it. New and updated pages can wait days or weeks before Google or Bing visits them. Submit the page to send it to Google, and add Bing to send it to Bing as well. It works for any public page, including articles, reviews and posts on other sites that mention you. See which search engines to submit to, submitting to Google and submitting to Bing, or check the page can be indexed first.