---
title: "AI crawler checker for ChatGPT, Google, Bing and Perplexity"
description: "Check which search and AI crawlers your robots.txt allows: Googlebot, Bingbot, OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot and more. Free to use."
canonical: "https://indexmysite.app/tools/ai-crawler-checker"
language: "en"
---

# Which search and AI crawlers can read your page?

Paste a page address. We read the site's robots.txt and show whether Google, Bing, ChatGPT, Perplexity, Claude and Apple can crawl that page, and which rule decides it.

## How to use this tool

Enter a public page address and start the check. After a short security check, the tool reads the robots.txt file for that site and lists each crawler as allowed or blocked for that page, with the rule that decides it. It reads only robots.txt.

## AI answers start with crawling and indexing

An AI answer can only link to a page its search system has crawled. Google says a page must be indexed and eligible for a snippet to appear as a link in AI Overviews or AI Mode, and the site must not be opted out in Search Console. ChatGPT, Perplexity and Claude run their own search crawlers. If robots.txt blocks one of them, that product cannot crawl your page for its answers, however good the page is.

Training crawlers are different. Blocking GPTBot, ClaudeBot or Google-Extended keeps your content out of model training. Your pages stay in search and AI answers, so many sites block those and allow the rest. See [what each AI answer needs](https://indexmysite.app/ai-search-indexing) before it can cite a page.

## The crawlers this tool checks

### Search and AI answers

| Crawler | Used for | If you block it |
| --- | --- | --- |
| [`Googlebot`](https://indexmysite.app/tools/ai-crawler-checker/googlebot) | Google Search, AI Overviews and AI Mode | Google cannot crawl the page for Search, including AI Overviews and AI Mode. [Source](https://developers.google.com/search/docs/appearance/ai-features) |
| [`Bingbot`](https://indexmysite.app/tools/ai-crawler-checker/bingbot) | Bing and Copilot | Bing cannot crawl the page for its index. [Source](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) |
| [`OAI-SearchBot`](https://indexmysite.app/tools/ai-crawler-checker/oai-searchbot) | ChatGPT search | The page is not shown in ChatGPT search answers, apart from navigational links. [Source](https://platform.openai.com/docs/bots) |
| `Claude-SearchBot` | Claude search | Claude does not index the page for search, which may reduce its visibility in answers. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| [`PerplexityBot`](https://indexmysite.app/tools/ai-crawler-checker/perplexitybot) | Perplexity search | Perplexity does not surface and link the page in its search results. [Source](https://docs.perplexity.ai/guides/bots) |
| [`Applebot`](https://indexmysite.app/tools/ai-crawler-checker/applebot) | Siri, Spotlight and Safari | Apple cannot crawl the page for search across its devices. [Source](https://support.apple.com/en-us/119829) |

### Visits a person asks for

| Crawler | Used for | If you block it |
| --- | --- | --- |
| `ChatGPT-User` | ChatGPT, when a person asks it to visit a page | OpenAI says robots.txt rules may not apply to these visits. [Source](https://platform.openai.com/docs/bots) |
| `Claude-User` | Claude, when a person asks it to visit a page | Claude cannot fetch the page to answer a question. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| `Perplexity-User` | Perplexity, when a person asks it to visit a page | Perplexity says these visits generally ignore robots.txt. [Source](https://docs.perplexity.ai/guides/bots) |

### Model training only

| Crawler | Used for | If you block it |
| --- | --- | --- |
| [`GPTBot`](https://indexmysite.app/tools/ai-crawler-checker/gptbot) | OpenAI model training | Content is not used to train OpenAI models. ChatGPT search is unaffected. [Source](https://platform.openai.com/docs/bots) |
| [`ClaudeBot`](https://indexmysite.app/tools/ai-crawler-checker/claudebot) | Anthropic model training | Future content is left out of Anthropic training data. Claude search is unaffected. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| [`Google-Extended`](https://indexmysite.app/tools/ai-crawler-checker/google-extended) | Gemini training and grounding | Gemini apps do not train on or ground answers with the page. Google Search, including AI Overviews, is unaffected. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) |

Applebot follows your Googlebot rules when no group names Applebot, so a Googlebot block also blocks Apple's search.

## A robots.txt that keeps you in AI answers

This blocks the three training crawlers and allows everything else. A crawler named in a group follows only that group, so if you name Googlebot or another search crawler anywhere in the file, repeat any shared Disallow lines in its group.

```
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /
```

## Allowed everywhere? Get the page indexed

Crawl rules let crawlers in. Submitting a page brings them to it. New and updated pages can wait days or weeks before Google or Bing visits them. [Submit the page](https://app.indexmysite.app/signup) to send it to Google, and add Bing to send it to Bing as well. It works for any public page, including articles, reviews and posts on other sites that mention you. See [which search engines to submit to](https://indexmysite.app/submit-website-to-search-engines), [submitting to Google](https://indexmysite.app/submit-website-to-google) and [submitting to Bing](https://indexmysite.app/submit-url-to-bing), or [check the page can be indexed](https://indexmysite.app/tools/indexability) first.

## Frequently asked questions

### Which crawlers decide whether I can appear in AI answers?

The search crawlers. Googlebot crawls for Google Search, and Google says a page must be indexed to be shown as a link in AI Overviews or AI Mode. OAI-SearchBot crawls for ChatGPT search, PerplexityBot for Perplexity, Claude-SearchBot for Claude, and Bingbot for Bing and Copilot. Block one and that product cannot crawl your page for its answers.

### If I block GPTBot, will ChatGPT stop showing my site?

No. GPTBot only collects content for training OpenAI models. ChatGPT search uses OAI-SearchBot, and OpenAI treats the two separately, so you can block training and still appear in ChatGPT search answers.

### Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search and follow your Googlebot rules. Google-Extended controls Gemini apps training and grounding.

### Why does it say a blocked crawler may still visit?

ChatGPT-User and Perplexity-User fetch a page when a person asks the assistant to read it. OpenAI says robots.txt rules may not apply to these visits, and Perplexity says its fetcher generally ignores them. Claude-User follows robots.txt.

### Does an allowed crawler mean my page is in its index?

No. Crawl rules only say who may visit. The crawler still has to find the page, and the search engine decides whether to index it. Submitting the page to Google and Bing helps their crawlers find it sooner.

### What does the checker read?

Only the robots.txt file at the root of the page's site. It does not open the page itself, so it cannot see a noindex tag or a firewall that blocks crawlers. Check the page with the indexability checker for those.
