Public URL scans, no accounts

AI Crawler Checker is designed as a no-login public URL scanner. The current service does not store scan reports by default, does not train models on submitted URLs, and does not crawl private pages.

What you submit

A submitted URL is used to fetch public robots.txt, the target public page, sitemap.xml, llms.txt, and llms-full.txt when present.

What not to submit

Do not submit private admin URLs, staging systems, secret links, or pages that require authentication. The scanner is intended for public web pages only.

Abuse and cost controls

The API route uses request timeouts, private-network blocking, redirect limits, response-size limits, and a lightweight rate limit. These controls protect the service and keep public scans conservative.

Future features

If saved reports, monitoring, or accounts are added later, this policy should be updated before launch.