Check public crawler policy
Parse robots.txt for major AI search, training, classic search, and common crawlers at the exact URL path.
AI crawler access check
Check If AI Crawlers Can Access Your Website
Test GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, robots.txt, headers, meta robots, sitemap, llms.txt, and user-triggered fetcher context in one transparent report.
What it checks
More than a robots.txt glance
A crawler policy can look open while a page still blocks discovery through headers, meta tags, redirects, thin rendered content, missing AI-readable discovery files, or confusing bot roles.
Inspect page-level signals
Review HTTP status, redirects, meta robots, X-Robots-Tag, canonical, readable text, and JSON-LD.
Find AI-readiness gaps
Check sitemap, llms.txt, llms-full.txt, user-triggered fetcher context, and copy-ready fixes without promising guaranteed AI visibility.
Boundaries
A check, not a guarantee
This tool diagnoses public access signals that site owners control. It does not bypass bot defenses, log into websites, or promise citation in any AI answer surface.
Visibility mode
Find accidental blocks that may prevent AI search or retrieval systems from reading public pages.
Protection mode
See which training or retrieval bots you are allowing, then choose a policy intentionally.
Evidence first
Every recommendation ties back to a visible rule, header, tag, status, missing file, or documented bot behavior.
Tool matrix
One scanner, focused readiness reports
Start with the full AI crawler access scan, then use the focused pages when the job is llms.txt validation, technical AEO readiness, or AI search visibility prerequisites.
llms.txt CheckerValidate llms.txt and llms-full.txt structure alongside sitemap, robots.txt, and page signals.AEO CheckerRun a technical Answer Engine Optimization readiness check with evidence-backed fixes.AI Search Visibility CheckerCheck technical visibility readiness before moving into prompt or citation tracking.Robots.txt AI CheckerInspect AI crawler policy at the exact submitted URL path.