robots.txt rules for AI crawlers
Choose which AI crawlers may read your site. Allow the ones that put you in AI answers, block training if you prefer, and copy the rules.
# AI search: let them in
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
# Training only: keep out
User-agent: GPTBot
Disallow: /# AI crawlers: generated with aisearchanalyzer.com/tools/robots-txt-generator # Allowed: these put you in AI answers or you chose to allow them User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / # Blocked User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Meta-ExternalAgent Disallow: / # Everyone else, including Googlebot and Bingbot User-agent: * Allow: /
Pick with a switch
Allow the crawlers that put you in AI answers. Block training crawlers if you'd rather.
Rules ready to paste
Copy a clean robots.txt block. No syntax to look up.
Where it goes
It lives at yourdomain.com/robots.txt. Most site builders have an editor for it.
This checks one thing. The free scan checks everything.
- What ChatGPT, Gemini, Claude, Perplexity and Google's AI say about you, with a free account
- 157 checks across up to 50 pages, scored in plain English
- Every fix in the order that matters, with steps for your platform
Questions
Can't find your answer? The help center covers every part of the report, or ask us directly.
Should I block AI crawlers?
Block training crawlers if you do not want your content used to train models. Keep search crawlers allowed if you want to appear in ChatGPT, Claude and Perplexity answers.
Where does robots.txt go?
At the root of your domain, so it loads at yourdomain.com/robots.txt. Most site builders have a robots.txt editor in their SEO settings.
Do all AI crawlers obey robots.txt?
The major ones from OpenAI, Anthropic, Google, Perplexity, Apple and Microsoft say they do. Firewalls are the way to stop the ones that do not.