ClaudeBot is not every Claude request
ClaudeBot, Claude-SearchBot, and Claude-User are shown separately so training, search retrieval, and user-triggered access do not collapse into one status.
Anthropic crawler policy
ClaudeBot Checker
Run a focused ClaudeBot access report and compare Anthropic crawler roles in one place. The checker separates ClaudeBot from Claude-SearchBot and Claude-User, then checks whether page-level signals make the URL usable.
Real checks
What this checker actually tests
The scan uses the same live crawler-access engine as the homepage, with the focused bot result lifted to the top of the report.
Robots rules are path-specific
The report evaluates the tested URL path, not only the homepage, and shows the matching robots.txt rule line when available.
Discovery blockers still matter
Even when ClaudeBot is allowed, noindex headers, missing sitemap signals, or thin readable HTML can reduce the practical value of access.
Methodology
How the bot decision is made
Each recommendation ties back to a public robots.txt rule, header, meta tag, file, or fetched page signal.
Evaluate Anthropic agents separately
The focused report checks ClaudeBot first, then shows Claude-SearchBot and Claude-User for role clarity.
Use exact robots matching
The scanner parses user-agent groups and Allow/Disallow rules for the submitted URL path.
Add page-readiness context
The result includes page fetch status, noindex directives, sitemap, and llms.txt because bot access alone is not a visibility guarantee.
Limitations
What this report does not claim
Bot access is a technical readiness signal, not a promise of training, ranking, citation, or AI visibility.
Boundary
This checker does not prove whether Anthropic has used, will use, or will cite the submitted page.
Boundary
Claude-User is shown for role clarity and is not scored the same way as an automatic crawler.
Boundary
The scan does not bypass login, WAF rules, paywalls, or private-network protections.
Tool matrix
Related checks in this suite
Move from one bot decision into broader robots, visibility, and crawler-readiness checks.
AI Search Visibility CheckerCheck technical discovery and readability foundations after Claude crawler access is clear.Full AI crawler checkRun the complete report across AI search, AI training, classic search, headers, sitemap, and llms.txt.Robots.txt AI CheckerReview AI crawler rules by user-agent and exact URL path.AI crawler listSee the maintained bot registry, official sources, categories, and robots.txt applicability.MethodologyRead how scans are bounded, scored, and kept privacy-first.
FAQ
Common questions
Short answers for site owners deciding how to handle crawler-specific robots.txt policies.
What does the ClaudeBot checker test?
It checks whether ClaudeBot is allowed or blocked by robots.txt for the exact submitted path, then reviews related Claude agents and page-level blockers such as noindex headers, sitemap, and llms.txt.
Is ClaudeBot the same as Claude-SearchBot or Claude-User?
No. This page separates ClaudeBot from Claude-SearchBot and Claude-User so training, search/retrieval, and user-triggered behavior can be reviewed independently.
Why does the page check headers and metadata too?
robots.txt access is only one layer. A page can be allowed for a crawler but still carry noindex directives or weak discovery signals.
Do you store ClaudeBot scan results?
No. The scanner is privacy-first and does not create public saved reports by default.