Claude can answer from what it learned in training, or search the web and read pages when a question needs current information. Anthropic uses a separate crawler for each job, and each one follows its own line in your robots.txt.
The three Claude crawlers
| Crawler | What Anthropic says it does | If you block it |
|---|---|---|
| ClaudeBot | Collects web content that may be used for training | Your future content is left out of training |
| Claude-SearchBot | Crawls the web to improve search results for users | Your site may show up less, and less accurately, in Claude’s search answers |
| Claude-User | Opens pages when someone asks Claude a question | Claude can’t read your page for that person |
The important line is the middle one. Many sites block “all AI bots” with a list copied from a forum, and that list often includes Claude-SearchBot. They meant to opt out of training and opted out of being found.
Allow search, decide on training
If you want Claude to recommend you but would rather not have your content in training, block ClaudeBot and allow Claude-SearchBot and Claude-User:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
The robots.txt AI rules tool writes this for every major AI company in one go.
Beyond robots.txt
Two more things decide whether Claude can use your site. Your server has to answer its crawlers with the real page, not a firewall challenge. And the content has to be in the HTML, because crawlers that build search indexes generally don’t run JavaScript. The AI crawler checker tests both.
Check what Claude says
Ask Claude about your category and see who it names. A free account asks Claude and four other AI tools the questions your customers ask, and fact-checks what they say about you against your site.