Search retrieval is distinct
Search-related bots may have a different role than training crawlers. Treat them as separate policy choices.
OpenAI search access
Inspect OAI-SearchBot access separately from GPTBot so search/retrieval policy and training policy stay clear.
Real checks
The scan uses the same live crawler-access engine as the homepage, with the focused bot result lifted to the top of the report.
Search-related bots may have a different role than training crawlers. Treat them as separate policy choices.
The scanner evaluates rules against the exact submitted path, not only the domain root.
Use targeted Allow or Disallow rules instead of broad patterns that create unintended blocks.