Does llms.txt help AI search? An honest answer

llms.txt is a plain text file at the root of a website, at /llms.txt, that lists your most important pages in Markdown with a short description of each. The idea, proposed in 2024, is to give AI tools a clean map of your site so they don’t have to dig through navigation and scripts.

It’s a reasonable idea. It’s also the most overrated item on most AI search checklists right now. Here’s why, and what to do instead.

What AI crawlers actually do with it

Several hosting and analytics companies have published their server logs since the file was proposed, and they tell the same story: AI crawlers rarely ask for /llms.txt. The big crawlers fetch robots.txt and your pages constantly, and fetch llms.txt on a small share of the sites they visit, if at all. Most llms.txt files get no requests in a given month.

That makes sense once you look at how AI answers are built. When ChatGPT, Perplexity or Google’s AI answers a question, it searches, fetches the pages that match, and reads them. Nothing in that loop starts with “check the site’s llms.txt first.”

What Google says

Google has been direct about it. Its search team has said Google doesn’t use llms.txt and has no plans to, and one of its advocates compared the file to the old keywords meta tag, which search engines stopped trusting years ago. Google’s AI Overviews and AI Mode draw on the same index as regular search, so for Google, the file changes nothing.

So should you have one?

Yes, if it takes you an hour. It’s a small forward bet: if a tool starts using it, you’re ready. Two rules if you do:

• Keep it true. A file that lists pages you’ve deleted or describes services you’ve dropped is worse than no file, because it’s a second, wrong description of your business.
• Don’t let it eat the time that matters. It’s a nice-to-have, not a strategy.

Our free llms.txt builder and validator builds one from your key pages, or checks the one you already have against the format, so the hour is closer to five minutes.

What actually gets you cited

The things that move AI answers are the same things that make a page easy to quote and easy to trust:

1. Let the crawlers in. Check that robots.txt and your firewall allow OpenAI’s, Anthropic’s, Perplexity’s and Google’s crawlers. A blocked crawler is the most common reason a business is invisible. Our AI crawler checker tests each one.
2. Put the answer in the HTML. Most AI crawlers don’t run JavaScript. If your page only shows its text after scripts load, they see an empty page. See JavaScript and AI crawlers.
3. Answer the question near the top. A short, direct answer in the first paragraph is the span an AI tool lifts into its reply.
4. Say who you are in structured data. Organization or LocalBusiness markup with your name, address, hours and links tells every engine the same facts. Our schema checker shows what’s there.
5. Keep pages fresh. Dated, updated content gets picked over stale pages for anything time-sensitive.
6. Be mentioned elsewhere. Reviews, directories, local guides and community threads are where AI tools learn that people recommend you. That work happens off your site.

The short version

llms.txt is harmless, cheap and mostly unread. Have one, keep it accurate, and spend the rest of your time on crawler access, readable pages, clear answers and structured data. A free scan checks all of those at once and shows what ChatGPT and the other AI tools say about you today.