Index My Site

Which search and AI crawlers can read your page?

Paste a page address. We read the site's robots.txt and show whether Google, Bing, ChatGPT, Perplexity, Claude and Apple can crawl that page, and which rule decides it.

We read the site's robots.txt file and nothing else. Crawl rules say who may visit. They do not show whether a page is indexed or cited.

Results

View robots.txt

AI answers start with crawling and indexing

An AI answer can only link to a page its search system has crawled. Google says a page must be indexed and eligible for a snippet to appear as a link in AI Overviews or AI Mode, and the site must not be opted out in Search Console. ChatGPT, Perplexity and Claude run their own search crawlers. If robots.txt blocks one of them, that product cannot crawl your page for its answers, however good the page is.

Training crawlers are different. Blocking GPTBot, ClaudeBot or Google-Extended keeps your content out of model training. Your pages stay in search and AI answers, so many sites block those and allow the rest. See what each AI answer needs before it can cite a page.

The crawlers this tool checks

Search and AI answers

CrawlerUsed forIf you block it
GooglebotGoogle Search, AI Overviews and AI ModeGoogle cannot crawl the page for Search, including AI Overviews and AI Mode. Source
BingbotBing and CopilotBing cannot crawl the page for its index. Source
OAI-SearchBotChatGPT searchThe page is not shown in ChatGPT search answers, apart from navigational links. Source
Claude-SearchBotClaude searchClaude does not index the page for search, which may reduce its visibility in answers. Source
PerplexityBotPerplexity searchPerplexity does not surface and link the page in its search results. Source
ApplebotSiri, Spotlight and SafariApple cannot crawl the page for search across its devices. Source

Visits a person asks for

CrawlerUsed forIf you block it
ChatGPT-UserChatGPT, when a person asks it to visit a pageOpenAI says robots.txt rules may not apply to these visits. Source
Claude-UserClaude, when a person asks it to visit a pageClaude cannot fetch the page to answer a question. Source
Perplexity-UserPerplexity, when a person asks it to visit a pagePerplexity says these visits generally ignore robots.txt. Source

Model training only

CrawlerUsed forIf you block it
GPTBotOpenAI model trainingContent is not used to train OpenAI models. ChatGPT search is unaffected. Source
ClaudeBotAnthropic model trainingFuture content is left out of Anthropic training data. Claude search is unaffected. Source
Google-ExtendedGemini training and groundingGemini apps do not train on or ground answers with the page. Google Search, including AI Overviews, is unaffected. Source

Applebot follows your Googlebot rules when no group names Applebot, so a Googlebot block also blocks Apple's search.

A robots.txt that keeps you in AI answers

This blocks the three training crawlers and allows everything else. A crawler named in a group follows only that group, so if you name Googlebot or another search crawler anywhere in the file, repeat any shared Disallow lines in its group.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Allowed everywhere? Get the page indexed

Crawl rules let crawlers in. Submitting a page brings them to it. New and updated pages can wait days or weeks before Google or Bing visits them. Submit the page to send it to Google, and add Bing to send it to Bing as well. It works for any public page, including articles, reviews and posts on other sites that mention you. See which search engines to submit to, submitting to Google and submitting to Bing, or check the page can be indexed first.

Frequently asked questions

The search crawlers. Googlebot crawls for Google Search, and Google says a page must be indexed to be shown as a link in AI Overviews or AI Mode. OAI-SearchBot crawls for ChatGPT search, PerplexityBot for Perplexity, Claude-SearchBot for Claude, and Bingbot for Bing and Copilot. Block one and that product cannot crawl your page for its answers.

No. GPTBot only collects content for training OpenAI models. ChatGPT search uses OAI-SearchBot, and OpenAI treats the two separately, so you can block training and still appear in ChatGPT search answers.

No. Google says Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search and follow your Googlebot rules. Google-Extended controls Gemini apps training and grounding.

ChatGPT-User and Perplexity-User fetch a page when a person asks the assistant to read it. OpenAI says robots.txt rules may not apply to these visits, and Perplexity says its fetcher generally ignores them. Claude-User follows robots.txt.

No. Crawl rules only say who may visit. The crawler still has to find the page, and the search engine decides whether to index it. Submitting the page to Google and Bing helps their crawlers find it sooner.

Only the robots.txt file at the root of the page's site. It does not open the page itself, so it cannot see a noindex tag or a firewall that blocks crawlers. Check the page with the indexability checker for those.