Separate Claude crawler roles
Claude crawler names can represent different purposes. A useful policy separates training access from search and user-triggered retrieval.
Anthropic crawler policy
ClaudeBot Checker
Check ClaudeBot, Claude-SearchBot, and Claude-User rules alongside page-level signals that can affect public crawlability.
Watch headers and meta tags
Even when robots.txt allows access, noindex headers or page metadata can make a page unsuitable for discovery.
Keep policy readable
The checker helps surface the exact user-agent group and path rule that applies to the tested URL.