OAI-SearchBot
Search and retrieval for OpenAI search experiences.
Operator: OpenAI
Category: AI Search
User-agent: OAI-SearchBot
Robots.txt: Scored
Official source
Updated 2026-05-15
AI crawler user agents checked by the tool
ChatGPT-User
User-triggered visits from ChatGPT and Custom GPT actions; not an automatic web crawler.
Operator: OpenAI
Category: User Triggered
User-agent: ChatGPT-User
Robots.txt: Not scored: may not apply
Official source
GPTBot
OpenAI web crawler used for model improvement according to OpenAI documentation.
Operator: OpenAI
Category: AI Training
User-agent: GPTBot
Robots.txt: Scored
Official source
Claude-SearchBot
Search and retrieval crawler for Claude experiences.
Operator: Anthropic
Category: AI Search
User-agent: Claude-SearchBot
Robots.txt: Scored
Official source
Claude-User
User-triggered fetches from Claude.
Operator: Anthropic
Category: User Triggered
User-agent: Claude-User
Robots.txt: Not scored: may not apply
Official source
ClaudeBot
Anthropic crawler for model-related web access.
Operator: Anthropic
Category: AI Training
User-agent: ClaudeBot
Robots.txt: Scored
Official source
PerplexityBot
Perplexity crawler for search and answer experiences.
Operator: Perplexity
Category: AI Search
User-agent: PerplexityBot
Robots.txt: Scored
Official source
Perplexity-User
User-triggered fetches used when Perplexity answers a user request.
Operator: Perplexity
Category: User Triggered
User-agent: Perplexity-User
Robots.txt: Not scored: generally ignored
Official source
Google-Extended
Google product improvement control token for Gemini and Vertex AI training use.
Operator: Google
Category: AI Training
User-agent: Google-Extended
Robots.txt: Scored
Official source
Googlebot
Classic Google Search crawling and indexing.
Operator: Google
Category: Classic Search
User-agent: Googlebot
Robots.txt: Scored
Official source
Bingbot
Classic Bing Search crawling and indexing.
Operator: Microsoft
Category: Classic Search
User-agent: bingbot
Robots.txt: Scored
Official source
CCBot
Common Crawl web archive crawler used by many downstream projects.
Operator: Common Crawl
Category: Common AI
User-agent: CCBot
Robots.txt: Scored
Official source
Applebot-Extended
Apple extension token for AI-related use controls.
Operator: Apple
Category: AI Training
User-agent: Applebot-Extended
Robots.txt: Scored
Official source
Meta-ExternalAgent
Meta crawler token commonly used for AI training controls.
Operator: Meta
Category: AI Training
User-agent: Meta-ExternalAgent
Robots.txt: Scored
Official source
Bytespider
ByteDance crawler often discussed in AI crawler policies.
Operator: ByteDance
Category: AI Training
User-agent: Bytespider
Robots.txt: Scored
Official source