GPTBot Checker

1. Choose training and search policies separately

OpenAI assigns training to GPTBot and search to OAI-SearchBot. The example opts out of the former while permitting the latter. ChatGPT-User handles user requests, where robots.txt may not apply. Read the operator’s crawler documentation.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Merge the groups you want into the existing root /robots.txt; preserve unrelated policies and sitemap declarations. Test a representative public page and a path that should stay restricted. Each hostname needs its own policy. To permit training instead, use Allow: / in the GPTBot group and review any more specific Disallow rules. robots.txt is not authentication for private content.

2. Investigate a Cloudflare block

1. Record the failing URL, timestamp, HTTP status, and request or Ray ID. Compare the exact host and path with the checker’s matched rule.
2. In Cloudflare Security Settings, review the AI bot policies for Training, Search, and Agent activity, plus any matching custom WAF rules. Keep the categories aligned with your intended robots policy.
3. Inspect the corresponding security event. Change only the rule responsible for an unwanted block, then check a new request. A broad WAF bypass based on a user-agent string also admits spoofed traffic.

Dashboard controls and defaults can change. Use Cloudflare’s current AI bot policy instructions. An HTTP 200 from this tool only describes its own request; official bots can receive a different response.

3. Detect a claimed bot, then verify its source IP

Filter your access logs for GPTBot in the user-agent field. Retain the client IP, time, method, path, status, and bytes served. A matching string is a claim anyone can send. Compare the actual client IP with the operator’s current published IP ranges using CIDR membership, not text-prefix matching. Behind a proxy, use its authenticated client-IP field; do not trust an arbitrary forwarded header from a public request.

# Save as verify_bot_ip.py; use a client IP from your trusted edge logs.
import ipaddress, json, sys, urllib.request

address = ipaddress.ip_address(sys.argv[1])
with urllib.request.urlopen("https://openai.com/gptbot.json", timeout=10) as response:
    prefixes = json.load(response)["prefixes"]
ranges = [ipaddress.ip_network(value) for item in prefixes
          for key, value in item.items() if key in ("ipv4Prefix", "ipv6Prefix")]
print("In published ranges" if any(address in network for network in ranges)
      else "Not in current published ranges")

Use the separate OAI-SearchBot range list when investigating OpenAI search traffic; GPTBot’s list verifies the training crawler’s network. A verified request with a successful response supports a visit claim. It does not establish training use, indexing, or citation.

4. Retest and keep the evidence

Run this URL check after publishing the rule, copy the report with its timestamp, and compare it with later edge logs. If the rule now permits access but requests still fail, investigate HTTP or WAF behavior. If requests succeed but no answer cites your page, continue with the visibility measurement checklist.