To check that a request saying ClaudeBot is really from Anthropic, match its IP against claude.com/crawling/bots.json. Anthropic says its bots follow robots.txt and pace their requests. It also says they do not try to get past CAPTCHAs, the puzzles that test for humans. In my logs for the week ending 09/09/2026, 11 of 12 ClaudeBot claims checked out.

Does blocking ClaudeBot remove pages it already collected?

Anthropic's help page does not say. It says a block signals that your future pages should stay out of its training data (checked 10/10/2026). It names no step for pages ClaudeBot collected before the block.

Do I still need anthropic-ai and Claude-Web in robots.txt?

No. Anthropic's current help page lists neither name, so they do nothing for its current bots. My file still lists anthropic-ai, and it does no harm there. What matters is that ClaudeBot, Claude-User and Claude-SearchBot each get the rule you want.

ClaudeBot: What It Crawls and How to Allow or Block It

The short answer

ClaudeBot is Anthropic’s training crawler. A crawler is a program that visits web pages automatically, and this one copies public pages so future Claude models can learn from them. Anthropic says it obeys robots.txt, the file on your site that tells crawlers what they may visit (checked 10/10/2026).

Seeing ClaudeBot in your server logs is normal. You do not need to do anything. You can check any visit against Anthropic’s published address list, which I show below.

Block it only if you want your future pages out of Anthropic’s training data. Live Claude answers come from two other bots, Claude-User and Claude-SearchBot, so leave those open if you want Claude to read and cite you. Anthropic’s help page says nothing about pages ClaudeBot collected before a block.

On 08/18/2026 I scanned 1,207 agency websites and filed ClaudeBot under crawlers that feed live AI answers. That one wrong label turned a choice about AI training into a fake emergency, and it is why this post sorts crawlers by job.

Anthropic's agents and what blocking each one does

Turn on the training-only block to see which agents it stops.

Dimmed: ClaudeBot. A training-only block stops only this agent. The rest keep their access.

Agent	Purpose	Honors robots.txt	Effect of blocking on training	Effect of blocking on live answers
ClaudeBotTrainingBlocked by this file	Purpose Collects web content that could contribute to model training	Honors robots.txt Yes. Anthropic says its bots honor robots.txt and support Crawl-delay	Effect of blocking on training Tells Anthropic to keep your future pages out of training data	Effect of blocking on live answers None described by Anthropic
Claude-SearchBotSearch index	Purpose Visits pages to make search results better for Claude users	Honors robots.txt Yes, same statement covers all Anthropic bots	Effect of blocking on training Not described by Anthropic	Effect of blocking on live answers Stops search indexing. Your pages may show up less, and less accurately, in user search results
Claude-UserUser fetch	Purpose Fetches a page when a Claude user asks a question that needs it	Honors robots.txt Yes, same statement covers all Anthropic bots	Effect of blocking on training Not described by Anthropic	Effect of blocking on live answers Stops Claude from fetching your page for a user's question. You may show up less when users search the web through Claude
anthropic-ai / Claude-Web (older names)	Purpose Not on Anthropic's current help page	Honors robots.txt Not listed	Effect of blocking on training Not listed	Effect of blocking on live answers Not listed

Every cell: Anthropic Help Center, checked 10/10/2026. Marked cells mean yes, or a described effect.

Anthropic links ClaudeBot to training only. The bots that touch live Claude answers are Claude-User and Claude-SearchBot.

Source: Anthropic Help Center, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (updated 04/07/2026) · 10/10/2026

What is ClaudeBot?

ClaudeBot is the crawler Anthropic uses to gather web pages that could help train its models. That is how Anthropic’s help page on its crawlers describes it (dated 04/07/2026, checked 10/10/2026).

Anthropic changed its crawler names this year, and PPC Land, a marketing news site, reported the change on 02/25/2026. Older names such as anthropic-ai and Claude-Web still show up in robots.txt files, but Anthropic’s current page lists neither. Searches for “claudebot” also mix in Clawdbot, an unrelated open-source assistant now called OpenClaw. Its current page names three bots, which it calls agents, and the next section covers each one.

What is the difference between ClaudeBot, Claude-User and Claude-SearchBot?

ClaudeBot gathers training data. Claude-SearchBot builds the index Claude searches, which is a stored list of pages it can look up fast. Claude-User fetches a page because a person asked Claude something right now.

This is the split between training and retrieval. Training means a model learns from your page for some future version, while retrieval means Claude fetches your page today for one person.

Anthropic spells out the cost of blocking each one. Blocking Claude-User stops Claude from retrieving your content for a user’s question. Blocking Claude-SearchBot stops Anthropic from indexing your content for search, and Anthropic says that may make you less visible in user search results.

Blocking ClaudeBot only signals that your future pages should stay out of training. So I decide by what each crawler does with my page, and the company behind it matters less. I think this split is the most useful thing I know about AI crawlers.

Cloudflare, a company whose network sits in front of many websites, sorts these bots the same way. Its AI Crawl Control bot list (updated 04/23/2026) files ClaudeBot under AI Crawler. Claude-SearchBot sits under AI Search, and Claude-User sits under AI Assistant.

What is the ClaudeBot user agent?

A user agent is the name a crawler gives when it visits your site. ClaudeBot’s user agent is a long line of text with the short name ClaudeBot inside it. Robots.txt rules match on that short name, which the standard calls the product token.

Anthropic’s page names the tokens but prints no full strings. Crawler directories, which are websites that list known bots, showed these on 10/10/2026.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +searchbot@anthropic.com)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)

Match on the token. The full strings differ between directories. A PPC Land list from 06/28/2026 shows a shorter ClaudeBot string, for example.

Anyone can fake a user agent. So the only test I trust is the IP (Internet Protocol) address the request came from. Anthropic publishes its crawler addresses at claude.com/crawling/bots.json, and on 10/10/2026 it held 26 blocks of IPv4 addresses.

I run this check on citability.dev, the site where I test how AI tools see the web. In the seven days ending 09/09/2026, 11 of 12 requests that claimed to be ClaudeBot came from a listed address. Only 7 of 19 Claude-User claims did.

I call the other 12 Claude-User claims unverified. I do not call them fakes, because Anthropic never promises that every fetch uses a listed address. My numbers check claims against a list. They prove nothing about citations or traffic.

Why does my site block ClaudeBot when I never chose to?

In my scans, most sites that block ClaudeBot never chose to, because the rule came from their content delivery network (CDN). A CDN sits in front of a site to speed it up and screen its traffic, and Cloudflare is one. I know this because I misread it at scale.

On 08/18/2026 my scan of 1,207 agency websites treated ClaudeBot blocks as if they hid sites from live AI answers. My code that read each robots.txt was fine, but the bug was a fixed list of crawlers that feed live answers. I had put ClaudeBot on it by mistake, next to OpenAI’s training crawler, GPTBot.

The tell was boring. Unrelated domains showed blocks at the exact same line numbers in their robots.txt. That usually means one tool wrote the file for all of them.

That tool is Cloudflare’s managed robots.txt, a feature that writes robots.txt rules for you. I checked 38 of the 1,207 domains, and 31 of them served that file. The real count of sites blocking a live-answer crawler was 2 of 1,207. Now I search for “Cloudflare Managed” before I make any crawler claim.

Cloudflare’s docs (updated 10/08/2026) show the managed file blocking eight crawlers, ClaudeBot among them, but not Claude-User or Claude-SearchBot. It also adds a content signal, a line that says yes to search and no to AI training. For Anthropic, that makes it a block-training-only file that the owners in my set never picked.

Cloudflare puts this section at the top of your robots.txt when it serves the file. So the copy you edit does not show it. Only the live file does, and editing your copy will not remove it, because Cloudflare regenerates that section.

Cloudflare added a second layer this year, which I covered in my Cloudflare September 15 post. Its changelog of 07/01/2026 set new defaults for domains that join Cloudflare from 09/15/2026. On those domains it blocks bots it files as Training or Agent on pages that show ads. Search bots stay allowed. The changelog does not say that existing domains change.

Cloudflare describes Agent traffic as real-time activity on a person’s behalf. So I would expect Claude-User to count as an Agent, but Cloudflare does not say so by name. If your pages carry ads, Claude may not get to read the page a user asked about.

The third layer is rate limiting, which caps how many requests one visitor can make in a short time. On 07/23/2026 I sent 12 fast requests with the ClaudeBot user agent to a site that looked like it blocked ClaudeBot. Nine got a 429, the error for too many requests.

A plain Chrome user agent got 12 of 12 through that minute, and bot requests spaced 25 seconds apart passed 14 of 15 times. So my honest word was throttling. Not blocking.

A firewall can also drop a bot before it reaches your server. Then your logs show nothing.

If your site is on Cloudflare, check four things before you edit your own file.

Fetch your live robots.txt and search it for “Cloudflare Managed”.
In Cloudflare, open Security Settings, filter by Bot traffic, and open Configure AI bot policies.
If your domain joined on or after 09/15/2026, check the ad-page defaults.
Look for rate limiting or firewall rules that match bot user agents.

Cloudflare’s managed robots.txt docs give that menu path (checked 10/10/2026).

Blocking everything vs blocking training only

Changed lines are shaded.

Blocks everything

User-agent: *Disallow: /admin/User-agent: ClaudeBotUser-agent: Claude-SearchBotUser-agent: Claude-UserDisallow: /

Blocks training only

User-agent: *Disallow: /admin/User-agent: ClaudeBotDisallow: /# Copy every Disallow line from your * group here.# /admin/ is only an example.User-agent: Claude-SearchBotUser-agent: Claude-UserDisallow: /admin/Allow: /

What each agent can do with the shown file

ClaudeBot
Blocks everything:Blocked. Tells Anthropic to keep your future pages out of training
Blocks training only:Blocked. Tells Anthropic to keep your future pages out of training
Claude-SearchBot
Blocks everything:Blocked. Your pages stay out of Claude search results
Blocks training only:Allowed. Your pages can show up in Claude search results
Claude-User
Blocks everything:Blocked. Claude cannot fetch your page when a user asks
Blocks training only:Allowed. Claude can fetch your page when a user asks

Python 3 urllib.robotparser read both files on 10/10/2026. Its can_fetch results match the outcomes above. No Anthropic agent could reach /admin/ in either file.

Both files stop ClaudeBot from taking training data. Only the first one also cuts Claude off from your pages in live answers.

Source: Anthropic Help Center (updated 04/07/2026); parsed with Python urllib.robotparser · 10/10/2026

If you are not sure which Anthropic agents your site turns away, run your domain through the free AEO audit. AEO (answer engine optimization) means making your site easy for AI answer tools to read and cite. You will see what your site tells AI crawlers before you edit a single line.

Should I block ClaudeBot?

Block ClaudeBot only if you want your future pages out of Anthropic’s training data. Leave Claude-User and Claude-SearchBot open if you want Claude to cite you. That makes it a licensing choice, a call about who may learn from your work. It is not a visibility fix.

I think publishers who sell their writing have a real argument for blocking training. My situation is the opposite. I want models to learn who I am and what I build. So I allow ClaudeBot on chudi.dev.

One mistake my audit notes flag is a blanket “block all AI bots” template. It blocks the training crawler, which is a fair choice. But it also blocks Claude-User and Claude-SearchBot, the agents that put your page in front of a person asking Claude.

If you want out of AI training in general, Anthropic is only one company. Cloudflare’s managed file, for example, also blocks GPTBot, Google-Extended and CCBot. My llms.txt and robots.txt guide covers crawlers from other companies. If you came here after a traffic drop, read why ChatGPT is not citing your website next, because the same split applies to OpenAI’s bots.

How do I allow or block ClaudeBot in robots.txt?

To allow all three Anthropic agents, the simplest file never names them, so they follow your wildcard group. To treat one agent differently, give it its own group. A group is a set of User-agent lines followed by the rules for those bots. The line User-agent: * starts the wildcard group, the catch-all group for any bot.

Read this rule before you copy a file. Robots.txt has an official standard, a document called RFC 9309 (Request for Comments, the name for numbered internet standards). Under it, a crawler obeys the group that names its token, and it only falls back to the wildcard group when no group names it.

Say your wildcard group disallows /admin/, and you add ClaudeBot in its own group. ClaudeBot no longer sees the admin rule. That is the trap. So every named group below repeats the wildcard Disallow lines. The /admin/ line is a placeholder. Copy your own private paths in its place.

Allow all three Anthropic agents by name.

User-agent: *
Disallow: /admin/

# Copy every Disallow line from your * group here.
# /admin/ is only an example.
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /admin/
Allow: /

Block training only, and keep live answers open. If you are unsure, I would start here, because it only gives up training.

User-agent: *
Disallow: /admin/

User-agent: ClaudeBot
Disallow: /

# Copy every Disallow line from your * group here.
# /admin/ is only an example.
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /admin/
Allow: /

Block everything from Anthropic. Its Disallow: / line already covers your private paths, so it needs no copied lines.

User-agent: *
Disallow: /admin/

User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /

Python’s standard robots.txt parser read all three files on 10/10/2026. In each one, no Anthropic agent could reach /admin/, and each agent got the access its label promises. Without the copied lines, the parser let Claude-User and Claude-SearchBot into /admin/.

My own file takes the other route. It lists the Anthropic names inside the wildcard group, so they share my rules for private pages. This excerpt from 10/10/2026 leaves out the other crawler names and the Allow lines.

User-agent: *
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: anthropic-ai
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /downloads/

Anthropic’s bots also support Crawl-delay, a rule outside the standard that asks a bot to wait between requests. So you can slow ClaudeBot without blocking it. The same group rule applies, so copy your Disallow lines into this group too.

User-agent: ClaudeBot
Crawl-delay: 1
Disallow: /admin/

Anthropic also says to add your rule on every subdomain you want to opt out, such as blog.example.com. It warns against blocking by IP, because then its bots cannot read your robots.txt.

After any change, fetch the live file with curl (a command-line tool that downloads a page). My content intent signaling post covers rules for how AI may use a page once it can read it.