llms.txt Doesn't Do What You Think
The internet told me to add an llms.txt file. “It helps AI tools find your content.” “It’s like robots.txt but for LLMs.”
I went looking for evidence. Here’s what I found.
What it is
llms.txt is a proposed standard by Jeremy Howard (Answer.AI), published September 2024. A markdown file at your site root that provides LLM-friendly content — titles, summaries, links to key pages. The idea: help AI tools understand your site without parsing HTML.
The pitch makes sense. Context windows are limited. HTML is messy. Site authors know what matters. Let them curate.
The problem
No major AI platform has confirmed they use it.
Google’s John Mueller, June 2025:
“FWIW no AI system currently uses llms.txt… It’s super-obvious if you look at your server logs. The consumer LLMs / chatbots will fetch your pages — for training and grounding, but none of them fetch the llms.txt file.”
He compared it to the keywords meta tag — “this is what a site-owner claims their site is about… why not just check the site directly?”
Google’s Gary Illyes at Search Central Live: “Google doesn’t support LLMs.txt and isn’t planning to.”
The data
SE Ranking analyzed 300,000 domains. Key findings:
- Only 10% had an llms.txt file
- No correlation between llms.txt and AI citations
- Removing the llms.txt variable from their ML model improved accuracy — it was adding noise
Server log analysis of 1,000 domains over 30 days: GPTBot absent entirely. ClaudeBot, PerplexityBot — zero requests for llms.txt.
The nuance
Anthropic is interesting. They haven’t officially confirmed Claude reads llms.txt, but they asked Mintlify to implement it for their docs. They maintain llms.txt on docs.anthropic.com.
But maintaining one and reading others’ are different things. Anthropic’s official crawler docs mention only robots.txt.
The summary
| Platform | Official support | Evidence |
|---|---|---|
| No — explicitly rejected | Mueller, Illyes statements | |
| OpenAI | No statement | No documentation |
| Anthropic | No statement | Uses internally, no confirmation Claude reads others’ |
| Perplexity | No statement | Has own file, no announcement |
The punchline
844,000+ sites have implemented llms.txt. The evidence says AI crawlers don’t request it.
I’m adding one anyway. It took five minutes, and if adoption ever tips, I’ll be ready.
The boring advice still applies: clear structure, good HTML semantics, useful content. There’s no shortcut file.