Robots.txt is path-specific
A rule that looks harmless at the domain level can block a product page, blog post, or documentation path.
Robots policy
Robots.txt AI Checker
Parse robots.txt for major AI user agents and see which rules apply to the URL path you care about.
Real checks
What this checker actually tests
The scan uses the same live crawler-access engine as the homepage, with the focused bot result lifted to the top of the report.
AI bots have different purposes
Training crawlers, search/retrieval crawlers, user-triggered fetchers, and classic search bots should be reviewed separately.
Rules need maintenance
AI crawler user-agent names change over time, so the bot registry should have a visible update date.