How the crawler access report works

AI Crawler Checker reviews public signals that site owners control. It starts with robots.txt, checks each supported bot at the submitted URL path, then inspects page-level metadata and discovery files.

What the score can tell you

The report can show that a public page is blocked by robots.txt, noindex metadata, X-Robots-Tag headers, a failed HTTP response, a redirect problem, missing readable text, missing structured data, or absent discovery files such as sitemap.xml and llms.txt.

What it cannot promise

Passing these checks does not guarantee AI visibility, ranking, citation, or inclusion in any answer engine. AI platforms change behavior over time and may use their own quality, safety, freshness, and source-selection systems.

Access boundary

The scanner only checks public URLs and public site settings. It does not bypass logins, paywalls, firewall rules, bot defenses, or private systems.