Check public crawler policy
Parse robots.txt for major AI search, training, classic search, common crawlers, and Content-Signal declarations at the exact URL path.
AI crawler access check
Detect GPTBot, ClaudeBot & AI Crawler Access
See which AI crawlers your robots.txt allows, check whether a public page responds, and find practical fixes. Compare GPTBot, ClaudeBot, and other search and training bots in one report.
What it checks
Review policy, access, and discovery
A page’s robots rules, HTTP response, and indexing directives answer different questions. Review them together before changing your crawler policy.
Inspect page-level signals
Review HTTP status, redirects, meta robots, X-Robots-Tag, canonical, readable text, and JSON-LD.
Choose your next fix
Review sitemap discovery, optional llms.txt files, and suggested fixes for the issues found.
Your crawl policy
Choose what your site allows
Set search and training permissions to suit your site. Use your access logs to verify visits; check AI answers separately for citations.
Allow discovery
Find accidental blocks that may prevent AI search or retrieval systems from reading public pages.
Manage training permissions
See which training or retrieval bots you are allowing, then choose a policy intentionally.
Verify a change
Retest the same URL after updating a rule, then compare the result with requests in your server or CDN logs.
Related tools
Choose a focused check
Start with the full AI crawler access scan, then use the focused pages when the job is llms.txt validation, technical AEO readiness, or AI search visibility prerequisites.
AI Crawler ListReview documented user agents, source links, categories, and robots.txt applicability.llms.txt CheckerValidate llms.txt and llms-full.txt structure alongside sitemap, robots.txt, and page signals.AEO CheckerRun a technical Answer Engine Optimization readiness check with evidence-backed fixes.AI Search Visibility CheckerCheck technical visibility readiness before moving into prompt or citation tracking.Robots.txt Checker for AI CrawlersInspect AI crawler policy at the exact submitted URL path.
Tool choice
Use the right check for the job
AI crawler access is one layer. Compare it with visibility tracking and llms.txt validation so teams do not treat one passing score or one Content-Signal line as a full AI search strategy.
Robots policy checkers
Best for confirming whether a specific user agent is allowed or blocked at one URL path.
AI visibility platforms
Best after technical blockers are fixed, when teams need prompt sampling, citations, and share-of-answer tracking.
llms.txt validators
Best for checking AI-readable discovery files, but they still need robots.txt, sitemap, metadata, and readable pages around them.
Crawler tokens vs use controls
Best for separating actual crawl access, such as Applebot, from AI-use control tokens, such as Applebot-Extended.
Content-Signal declarations
Best for spotting search, AI input, AI training, and optional immediate/reference/full use preferences published alongside crawler rules.