Many sites block AI crawlers to protect their content, and some block far more than they meant to. The difference matters: some crawlers collect pages to train models, others fetch pages so an AI can cite them in an answer. Block the second kind and you disappear from those answers.
Search crawlers: block these and AI cannot cite you
| Crawler | Company | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search answers |
| ChatGPT-User | OpenAI | Opens a page when a ChatGPT user asks it to |
| Claude-SearchBot | Anthropic | Finds pages for Claude’s search answers |
| Claude-User | Anthropic | Opens a page when a Claude user asks it to |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers |
| Perplexity-User | Perplexity | Opens a page when a Perplexity user asks it to |
| Googlebot | Google Search, AI Overviews and Gemini’s live answers | |
| Bingbot | Microsoft | Bing and Copilot; ChatGPT search draws on Bing too |
Training crawlers: blocking these is a business choice
| Crawler | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects pages to train OpenAI models |
| ClaudeBot | Anthropic | Collects pages to train Claude |
| Google-Extended | Controls use of your pages for Gemini training; does not affect Search | |
| Applebot-Extended | Apple | Controls use of your pages for Apple’s models |
| CCBot | Common Crawl | Builds an open dataset many models train on |
| Meta-ExternalAgent | Meta | Collects pages for Meta’s models |
Blocking training crawlers does not remove you from AI search answers. If you want to keep your content out of training but still be recommended, block the training crawlers and allow the search crawlers.
A robots.txt that does exactly that
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
The free robots.txt AI rules generator builds this for you with the crawlers you choose.
The block robots.txt cannot see
Some hosts, CDNs and security plugins block AI crawlers before robots.txt is ever read, often through a single “block AI bots” switch. Those switches frequently cover search crawlers as well as training ones. To check, request your homepage as a browser and as an AI agent and compare: the AI crawler checker does this in one step.
Quick checklist
- Load yoursite.com/robots.txt and look for Disallow: / under any search crawler above.
- Check your host, CDN and security plugin for an AI or bot blocking setting.
- Run the AI crawler checker to confirm the result from the outside.