Index My Site

ClaudeBot: Anthropic's training crawler and how to block it

ClaudeBot is the crawler Anthropic uses to collect web content for training its Claude models. Blocking it keeps your future pages out of that training. It does not control whether Claude search can find and cite you.

What ClaudeBot does

Anthropic says ClaudeBot collects web content that could contribute to training its generative AI models. When robots.txt blocks ClaudeBot, Anthropic treats that as a signal to leave the site’s future content out of its training data.

Claude’s search and its page visits use two other crawlers, Claude-SearchBot and Claude-User. Anthropic says blocking Claude-SearchBot may reduce a site’s visibility in Claude’s search results, so most sites that want to appear in Claude answers block ClaudeBot and allow the other two.

Anthropic supports the Crawl-delay rule in robots.txt, and asks you to add the rules to every subdomain you want to opt out.

OperatorAnthropic
robots.txt tokenClaudeBot
User agentAnthropic’s documentation names the ClaudeBot token but does not print a full user-agent string.
Used forAnthropic model training
If you block itFuture content is left out of Anthropic training data. Claude search is unaffected.
Check a visit is realAnthropic publishes its crawler IP addresses at claude.com/crawling/bots.json. A request from an address on that list comes from Anthropic. Anthropic warns that blocking its IP addresses may stop ClaudeBot reading your robots.txt, so use robots.txt to opt out.

Source: Anthropic: does Anthropic crawl data from the web

Check whether your robots.txt allows ClaudeBot

We read the site's robots.txt file and nothing else. Crawl rules say who may visit. They do not show whether a page is indexed or cited.

Results

View robots.txt

How to block or allow ClaudeBot in robots.txt

Use the ClaudeBot token in robots.txt. Anthropic says blocking by IP address may not work reliably, because it stops the crawler reading your robots.txt rules.

To block ClaudeBot from the whole site:

User-agent: ClaudeBot
Disallow: /

To allow it everywhere, even when another group blocks all crawlers:

User-agent: ClaudeBot
Allow: /

A crawler named in a group follows only that group, so repeat any shared Disallow lines in its group. See how robots.txt blocks work for the details.

Anthropic's other crawlers

CrawlerUsed forIf you block it
Claude-SearchBotClaude searchClaude does not index the page for search, which may reduce its visibility in answers.
Claude-UserClaude, when a person asks it to visit a pageClaude cannot fetch the page to answer a question.

Blocking ClaudeBot keeps you in search

ClaudeBot only collects training data. Claude-SearchBot is the crawler that indexes pages for Claude’s search, so leave it allowed if you want Claude to find and cite your pages. Google, Bing, ChatGPT and Perplexity have their own search crawlers, and blocking ClaudeBot changes none of them.

Frequently asked questions

Not on its own. ClaudeBot collects training data. Claude search uses Claude-SearchBot, and Anthropic says blocking that crawler may reduce your visibility in Claude’s search results.

Add a group for ClaudeBot to your robots.txt with Disallow: / and repeat it on each subdomain you want to opt out. Anthropic recommends robots.txt over IP blocking.

Source: Anthropic: does Anthropic crawl data from the web

Yes. Anthropic says it supports the non-standard Crawl-delay rule, for example Crawl-delay: 1.

Anthropic says a block signals that the site’s future materials should be left out of training datasets. It does not describe removing content collected earlier.