OAI-SearchBot
Search and retrieval for OpenAI search experiences.
Operator: OpenAI
Category: AI Search
User-agent: OAI-SearchBot
Robots.txt: Scored
Reference · Official IP ranges
Updated 2026-08-03
AI crawler user agents checked by the tool
Official IP-range links appear when the operator publishes them. Match both the user agent and current IP ranges when reviewing WAF rules or request logs.
OAI-AdsBot
Required crawler for ChatGPT Ads landing-page validation and review; not used for foundation-model training.
Operator: OpenAI
Category: Ad Validation
User-agent: OAI-AdsBot
Robots.txt: Scored
Reference · Official IP ranges
ChatGPT-User
User-triggered visits from ChatGPT and Custom GPT actions; not an automatic web crawler.
Operator: OpenAI
Category: User Triggered
User-agent: ChatGPT-User
Robots.txt: Not scored: may not apply
Reference · Official IP ranges
GPTBot
OpenAI web crawler used for model improvement according to OpenAI documentation.
Operator: OpenAI
Category: AI Training
User-agent: GPTBot
Robots.txt: Scored
Reference · Official IP ranges
Claude-SearchBot
Search and retrieval crawler for Claude experiences.
Operator: Anthropic
Category: AI Search
User-agent: Claude-SearchBot
Robots.txt: Scored
Reference
Claude-User
User-triggered fetches from Claude that site owners can control through robots.txt.
Operator: Anthropic
Category: User Triggered
User-agent: Claude-User
Robots.txt: Scored
Reference
ClaudeBot
Anthropic crawler for model-related web access.
Operator: Anthropic
Category: AI Training
User-agent: ClaudeBot
Robots.txt: Scored
Reference
PerplexityBot
Perplexity crawler for search and answer experiences.
Operator: Perplexity
Category: AI Search
User-agent: PerplexityBot
Robots.txt: Scored
Reference · Official IP ranges
Perplexity-User
User-triggered fetches used when Perplexity answers a user request.
Operator: Perplexity
Category: User Triggered
User-agent: Perplexity-User
Robots.txt: Not scored: generally ignored
Reference · Official IP ranges
Google-Extended
Google product improvement control token for Gemini and Vertex AI training use.
Operator: Google
Category: AI Training
User-agent: Google-Extended
Robots.txt: Scored
Reference
Google-CloudVertexBot
Google crawler token for crawls requested by site owners when building Vertex AI Agents; not a Google Search ranking signal.
Operator: Google
Category: Site-owner Requested
User-agent: Google-CloudVertexBot
Robots.txt: Scored
Reference
Googlebot
Classic Google Search crawling and indexing.
Operator: Google
Category: Classic Search
User-agent: Googlebot
Robots.txt: Scored
Reference
Bingbot
Classic Bing Search crawling and indexing.
Operator: Microsoft
Category: Classic Search
User-agent: bingbot
Robots.txt: Scored
Reference
CCBot
Common Crawl web archive crawler used by many downstream projects.
Operator: Common Crawl
Category: Common AI
User-agent: CCBot
Robots.txt: Scored
Reference
Applebot
Apple web crawler for search-style experiences such as Siri and Spotlight; Applebot-Extended controls AI-related use separately.
Operator: Apple
Category: Classic Search
User-agent: Applebot
Robots.txt: Scored
Reference
Applebot-Extended
Apple extension token for AI-related use controls.
Operator: Apple
Category: AI Training
User-agent: Applebot-Extended
Robots.txt: Scored
Reference
Meta-ExternalAgent
Meta crawler token commonly used for AI training controls.
Operator: Meta
Category: AI Training
User-agent: Meta-ExternalAgent
Robots.txt: Scored
Reference
Bytespider
ByteDance crawler often discussed in AI crawler policies.
Operator: ByteDance
Category: AI Training
User-agent: Bytespider
Robots.txt: Scored
Reference