Public evidence for crawler decisions

AI Crawler Checker is an independently operated public URL diagnostic project. It is not an official tool from OpenAI, Anthropic, Google, or Cloudflare and is not affiliated with those providers.

What we maintain

The site brings robots.txt decisions, page response signals, and discovery-file observations into one report. The operator maintains the scanner, the bot registry, and explanatory pages. The project does not sell a proprietary AI visibility score or claim access to answer-engine ranking systems.

Evidence and corrections

Bot roles and policy explanations link to operator documentation. The registry provides those sources and its latest update date. Scanner changes are checked against reproducible fixtures; live scans report a timestamp because a site’s responses and policies can change.

For a disputed result, retain the public URL, scan timestamp, matched robots rule and line, expected result, and the official documentation that supports it. Redact credentials, private paths, and unrelated log data before sharing evidence. A change to the site after a scan may explain a different result.

Recent maintenance

September 8, 2026: corrected encoded-path comparisons, added declared sitemap discovery and JSON-LD syntax diagnostics, separated optional llms files from baseline scoring, and expanded GPTBot/ClaudeBot verification guidance. These changes improve what the report can substantiate; they do not turn it into live bot-identity or citation tracking.

Read the methodology for the exact scope and scoring choices and privacy policy for scan, browser, analytics, and hosting data handling.