---
title: "ClaudeBot: Anthropic's training crawler and how to block it"
description: "ClaudeBot collects web content for training Anthropic models. What blocking it does, how it differs from Claude-SearchBot, and the robots.txt rules to use."
canonical: "https://indexmysite.app/tools/ai-crawler-checker/claudebot"
language: "en"
---

# ClaudeBot: Anthropic's training crawler and how to block it

ClaudeBot is the crawler Anthropic uses to collect web content for training its Claude models. Blocking it keeps your future pages out of that training. It does not control whether Claude search can find and cite you.

## What ClaudeBot does

Anthropic says ClaudeBot collects web content that could contribute to training its generative AI models. When robots.txt blocks ClaudeBot, Anthropic treats that as a signal to leave the site’s future content out of its training data.

Claude’s search and its page visits use two other crawlers, Claude-SearchBot and Claude-User. Anthropic says blocking Claude-SearchBot may reduce a site’s visibility in Claude’s search results, so most sites that want to appear in Claude answers block ClaudeBot and allow the other two.

Anthropic supports the Crawl-delay rule in robots.txt, and asks you to add the rules to every subdomain you want to opt out.

<table><tbody><tr><th scope="row">Operator</th><td>Anthropic</td></tr><tr><th scope="row">robots.txt token</th><td><code>ClaudeBot</code></td></tr><tr><th scope="row">User agent</th><td>Anthropic’s documentation names the ClaudeBot token but does not print a full user-agent string.</td></tr><tr><th scope="row">Used for</th><td>Anthropic model training</td></tr><tr><th scope="row">If you block it</th><td>Future content is left out of Anthropic training data. Claude search is unaffected.</td></tr><tr><th scope="row">Check a visit is real</th><td>Anthropic publishes its crawler IP addresses at claude.com/crawling/bots.json. A request from an address on that list comes from Anthropic. Anthropic warns that blocking its IP addresses may stop ClaudeBot reading your robots.txt, so use robots.txt to opt out.</td></tr></tbody></table>

Source: [Anthropic: does Anthropic crawl data from the web](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

## Check whether your robots.txt allows ClaudeBot

Use the [AI crawler checker](https://indexmysite.app/tools/ai-crawler-checker) to read a site's robots.txt and see whether ClaudeBot and the other search, AI and training crawlers may visit a page.

## How to block or allow ClaudeBot in robots.txt

Use the ClaudeBot token in robots.txt. Anthropic says blocking by IP address may not work reliably, because it stops the crawler reading your robots.txt rules.

To block ClaudeBot from the whole site:

```
User-agent: ClaudeBot
Disallow: /
```

To allow it everywhere, even when another group blocks all crawlers:

```
User-agent: ClaudeBot
Allow: /
```

A crawler named in a group follows only that group, so repeat any shared Disallow lines in its group. See [how robots.txt blocks work](https://indexmysite.app/blocked-by-robots-txt) for the details.

## Anthropic's other crawlers

| Crawler | Used for | If you block it |
| --- | --- | --- |
| `Claude-SearchBot` | Claude search | Claude does not index the page for search, which may reduce its visibility in answers. |
| `Claude-User` | Claude, when a person asks it to visit a page | Claude cannot fetch the page to answer a question. |

## Blocking ClaudeBot keeps you in search

ClaudeBot only collects training data. Claude-SearchBot is the crawler that indexes pages for Claude’s search, so leave it allowed if you want Claude to find and cite your pages. Google, Bing, ChatGPT and Perplexity have their own search crawlers, and blocking ClaudeBot changes none of them.

## Frequently asked questions

### Does blocking ClaudeBot stop Claude citing my site?

Not on its own. ClaudeBot collects training data. Claude search uses Claude-SearchBot, and Anthropic says blocking that crawler may reduce your visibility in Claude’s search results.

### How do I block ClaudeBot?

Add a group for ClaudeBot to your robots.txt with Disallow: / and repeat it on each subdomain you want to opt out. Anthropic recommends robots.txt over IP blocking.

Source: [Anthropic: does Anthropic crawl data from the web](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

### Does ClaudeBot follow Crawl-delay?

Yes. Anthropic says it supports the non-standard Crawl-delay rule, for example Crawl-delay: 1.

### Does ClaudeBot remove content that is already in a model?

Anthropic says a block signals that the site’s future materials should be left out of training datasets. It does not describe removing content collected earlier.
