Robots.txt is path-specific
A rule that looks harmless at the domain level can block a product page, blog post, or documentation path.
Robots policy
Robots.txt Checker for AI Crawlers
Parse robots.txt for major AI user agents and see which rules apply to the URL path you care about.
Included checks
What you can check
Review crawler rules, page responses, and discovery files for the URL you submit.
AI bots have different purposes
Training crawlers, search/retrieval crawlers, user-triggered fetchers, ad-validation fetchers, and classic search bots should be reviewed separately.
Rules need maintenance
AI crawler user-agent names change over time, so the bot registry should have a visible update date.