AI crawler
Claude-SearchBot
Crawlers that read pages to train or ground AI models.
Last updated
At a glance
| Category | AI crawler |
|---|---|
| Operator | Anthropic |
| User agent match | contains "claude-searchbot" |
| Executes JavaScript | No |
| Respects robots.txt | Generally yes |
| Identity verifiable | Yes, against published IP ranges |
| IVT class | GIVT (general invalid traffic: declared automation) |
What Claude-SearchBot is
Claude-SearchBot is an AI crawler: it reads pages so that an AI model can be trained on them or answer questions from them. It does not execute JavaScript, so a browser-based tracker never sees it; its visits exist only in server logs and in server-side collection.
Anthropic documents this bot, and the primary source beats any directory, ours included: official Claude-SearchBot documentation.
Can the name be trusted?
The operator publishes the IP ranges Claude-SearchBot calls from, and TrueVisit checks every visit claiming this name against them. A claim arriving from an address outside the published ranges fails verification and is reported as exactly that, so an impostor borrowing the name cannot borrow the trust.
Should you block it?
Blocking Claude-SearchBot is a trade, not hygiene. Crawlers like this one are how AI assistants read and then cite your site, so walling it off removes you from those answers while saving you its bandwidth. Measure it first, then decide: if the model behind it sends you cited readers, it is earning its keep; if it only takes, block it with a clear conscience.
User-agent: Claude-SearchBot
Disallow: /For enforcement rather than a polite request, the Cloudflare blocking guide generates copy-paste WAF rules from this same signature list, with guardrails that keep verified search engines safe.
How TrueVisit reports it
Visits from Claude-SearchBot appear in your bot report under this exact name, never as an anonymous percentage, and every verdict stores the reason it was reached. Because this client never executes JavaScript, a browser-only tracker cannot see it at all; it is visible through server-side collection, which is how the visits reach your report.