What ClaudeBot does
Anthropic says ClaudeBot collects web content that could contribute to training its generative AI models. When robots.txt blocks ClaudeBot, Anthropic treats that as a signal to leave the site’s future content out of its training data.
Claude’s search and its page visits use two other crawlers, Claude-SearchBot and Claude-User. Anthropic says blocking Claude-SearchBot may reduce a site’s visibility in Claude’s search results, so most sites that want to appear in Claude answers block ClaudeBot and allow the other two.
Anthropic supports the Crawl-delay rule in robots.txt, and asks you to add the rules to every subdomain you want to opt out.
| Operator | Anthropic |
|---|---|
| robots.txt token | ClaudeBot |
| User agent | Anthropic’s documentation names the ClaudeBot token but does not print a full user-agent string. |
| Used for | Anthropic model training |
| If you block it | Future content is left out of Anthropic training data. Claude search is unaffected. |
| Check a visit is real | Anthropic publishes its crawler IP addresses at claude.com/crawling/bots.json. A request from an address on that list comes from Anthropic. Anthropic warns that blocking its IP addresses may stop ClaudeBot reading your robots.txt, so use robots.txt to opt out. |