ClaudeBot

ClaudeBot is Anthropic's web crawler that collects public web content that could contribute to training its generative AI models. Anthropic runs two other bots with separate controls: Claude-SearchBot, which indexes pages for search results in Claude, and Claude-User, which fetches pages when a Claude user asks a question.

Why ClaudeBot gets so much attention

Anthropic's crawling became known for taking a lot and sending little back. Cloudflare measures a crawl-to-refer ratio: pages crawled for each visit a platform refers to a site. For Anthropic, it stood at about 38,000 in July 2025, the highest among major AI platforms, even after falling from about 286,900 in January. Cloudflare suggests referrals may be improving since web search arrived in Claude in March 2025, with clickable citations. In December 2024, Vercel counted 370 million requests from Anthropic's crawler across its network in a single month.

If you block ClaudeBot in response, the effect is precise and limited: it opts your future content out of Anthropic's training, and nothing more.

How Anthropic's three bots work

Anthropic documents three bots, each with its own robots.txt name:

ClaudeBot collects content that could contribute to model training. Restricting it "signals that the site's future materials should be excluded" from training datasets. The wording covers future materials, not content already collected.
Claude-SearchBot indexes content to improve search results in Claude. Blocking it "may reduce your site's visibility and accuracy in user search results."
Claude-User fetches pages when a person asks Claude a question. Blocking it stops Claude from retrieving your content for those requests.

Anthropic says its bots honor robots.txt, support the non-standard Crawl-delay directive, and need rules on every subdomain you want to cover. It warns that blocking its IP addresses instead may not work reliably, because it stops the bots from reading your robots.txt.

Example: robots.txt rules for Anthropic

Suppose a B2B software site wants out of training but still wants Claude to find its pages when a user asks "Which accounting tools support e-invoicing in France?" These rules express that:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
User-agent: Claude-User
Crawl-delay: 1
Allow: /

Claude-SearchBot and Claude-User can still index and fetch the site, slowed by Crawl-delay, a non-standard rule Anthropic supports, while ClaudeBot is told to stay out. Removing the second group would not block them: bots without a group of their own fall back to the User-agent: * group, if there is one.

How to manage ClaudeBot on your site

Decide per bot, not per company: training (ClaudeBot), search (Claude-SearchBot) and user requests (Claude-User) are three separate choices. Check your logs for each name, and confirm that the source IP appears on the list Anthropic publishes, since user agents are easy to fake. Then apply the same logic to OpenAI, Perplexity and Google, whose bots are compared in AI crawlers. Hawkscan does not track Claude, and its action plan does not check Anthropic's bots: it flags blocked AI answer crawlers such as OAI-SearchBot for ChatGPT search, and leaves ClaudeBot to your own policy.