Every WordPress site has a robots.txt, even if nobody ever made one. Open yoursite.com/robots.txt and you’ll see it. Look for it on your server and you won’t find it, because WordPress writes it on the fly each time it’s requested.
That catches a lot of people out: they search their files, find nothing, and upload a new one that quietly replaces whatever their SEO plugin was doing. Here is how it actually works, how to change it safely, and what to add so AI search tools can read your site.
Where is robots.txt in WordPress?
There’s no file by default. WordPress answers requests for /robots.txt itself and builds the text in code. You can see the result at your own address with /robots.txt on the end.
If a real robots.txt file exists in the site’s root folder (the one with wp-config.php in it), your web server sends that file instead, and WordPress’s version, along with anything a plugin adds to it, is never used.
What’s in the WordPress robots.txt by default
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/wp-sitemap.xml
Disallow: /wp-admin/keeps crawlers out of the dashboard.Allow: /wp-admin/admin-ajax.phplets them reach the one admin file some themes and plugins use to load content.- The
Sitemapline points to the sitemap WordPress builds (since version 5.5). SEO plugins often swap it for their own, such assitemap_index.xml.
That default is fine for most sites. Nothing in it blocks search engines or AI crawlers.
How to edit robots.txt in WordPress
Pick one way and stick with it. Mixing them is how rules go missing.
With an SEO plugin (easiest)
- Yoast SEO: Yoast SEO, then Tools, then File editor. This writes a real file to your server.
- Rank Math: Rank Math, then General Settings, then Edit robots.txt. This changes WordPress’s generated version.
- All in One SEO: All in One SEO, then Tools, then Robots.txt Editor. This also changes the generated version.
Menu names move between plugin versions, so look for “robots.txt” or “File editor” if yours differs.
With a real file
Make a plain text file named robots.txt, put your rules in it, and upload it to the site’s root folder with your host’s file manager or FTP. From then on, that file is what everyone gets. Plugin editors for the generated version stop having any effect.
In code
Developers can change the generated version with WordPress’s robots_txt filter in a small plugin or the theme’s functions.php. It keeps everything in version control and leaves no file to forget about.
The “Discourage search engines” setting
Under Settings, then Reading, WordPress has a box called “Discourage search engines from indexing this site.” Since WordPress 5.3 it doesn’t touch robots.txt at all. It adds a noindex tag to every page instead, which tells search engines, and the AI tools that rely on them, to leave the whole site out.
It’s meant for staging and unfinished sites. On a live site it should be unchecked. It’s one of the most common reasons a WordPress site never shows up anywhere, and it often gets left on after a launch.
robots.txt rules for AI crawlers
AI tools send their own crawlers, and they come in two kinds: search crawlers fetch pages so the AI can cite them in an answer, and training crawlers collect pages to train models. Blocking a training crawler doesn’t remove you from AI answers. Blocking a search crawler does.
To stay out of training but still be recommended, add this to your robots.txt, above or below the WordPress lines:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# AI search: let them in
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
# Training only: keep out
User-agent: GPTBot
Disallow: /If you’re happy for your pages to train models too, you don’t need any of this: the WordPress default already lets every crawler in. The full list of crawlers and what each one does is in which AI crawlers to allow, and the robots.txt generator writes the rules for the ones you choose.
- OAI-SearchBotSearchAllowed
- GPTBotTrainingAllowed
- ClaudeBotTrainingAllowed
- PerplexityBotSearchBlocked
- Google-ExtendedTrainingAllowed
robots.txt isn’t the only gate. Security plugins, your host and CDNs such as Cloudflare can block AI crawlers before robots.txt is ever read, often with one “block AI bots” switch that covers search crawlers too. The AI crawler checker requests your site as each crawler and tells you which ones get through.
Common robots.txt questions
How do I disallow everything?
User-agent: *
Disallow: /
This asks every crawler to stay out of the whole site. It’s rarely what you want on a live site. To keep a site out of search results, use the “Discourage search engines” setting or a noindex tag instead: a crawler that’s blocked by robots.txt can’t see a noindex tag, so a blocked page can still show up in results as a bare link.
Can I use noindex in robots.txt?
No. Google stopped supporting noindex lines in robots.txt in 2019, and other engines never did. Use a noindex meta tag on the page, which every WordPress SEO plugin can set per page.
Do I need the Sitemap line?
It’s not required, but it’s the easiest way to tell every crawler where your sitemap is. WordPress adds one for you; check that it still points to a sitemap that loads if you change SEO plugins.
Does Crawl-delay work?
Google ignores it. Bing respects it. If your server struggles with crawlers, a caching plugin usually does more good than slowing them down.
What about Shopify or other platforms?
Shopify also makes robots.txt for you, and lets you change it by adding a robots.txt.liquid template to your theme. Squarespace and Wix manage it for you with a few settings. The AI crawler rules above work the same everywhere.
Check yours in 30 seconds
Paste your address into the AI crawler checker to see your robots.txt as each AI crawler reads it, or run a free scan for the whole picture: robots.txt, the “Discourage search engines” tag, firewall blocks, and what ChatGPT says about your business.