Does llms.txt actually do anything for AI search?
Nothing published so far shows llms.txt affecting AI citations. According to Ahrefs, which studied 137,210 domains, 97% of the sites publishing an llms.txt got zero requests for that file across the whole of May 2026, and the bots that answer user queries accounted for 1.1% of requests to the files that were fetched at all. SE Ranking's study of roughly 300,000 domains found no relationship between having the file and citation frequency. Google's John Mueller said in June 2025 that no AI system uses it. Adoption is still climbing, which tells you how much of AEO is run on hope rather than measurement. The file is cheap, harmless and worth having if your buyers point coding agents at your documentation. It is not the reason anybody gets cited.
What is llms.txt supposed to do?
An llms.txt file is a plain markdown file at the root of a domain that lists the pages a site considers most useful to a language model, each with a short description. The proposal, published by Jeremy Howard of Answer.AI in 2024, treats the file as a curated map for machines: robots.txt sets crawl rules, sitemap.xml lists every URL, and llms.txt is meant to say which handful of pages actually matter and what each one covers.
The argument behind it is reasonable. A model working with a limited context window cannot read a whole site, and HTML pages carry navigation, cookie banners and scripts that waste tokens before the content starts. A markdown index pointing at clean markdown versions of the important pages would let an assistant fetch the right thing cheaply. Some proposals pair it with an llms-full.txt containing the entire documentation set as one file.
The gap is that nothing in the proposal obliges an engine to read the file. A standard only works when the consumers adopt it, and llms.txt arrived as a suggestion to companies that had not asked for one. Robots.txt succeeded because search engines agreed to honour it and publishers agreed to write it. Neither half of that has happened here on the retrieval side.
The distinction that matters when you decide whether to publish one is between two different consumers. A coding agent fetching your API docs on a developer's instruction is a reader you can help. An answer engine deciding which page to cite when a buyer asks about your category is a reader that has never asked for the file. Both get called AI, and only one of them is reading.
Do AI crawlers actually request llms.txt?
AI crawlers almost never request llms.txt, and the largest crawler-log study says so directly. Ahrefs analysed 137,210 domains, every domain in its web analytics product that received traffic in May 2026, and found that 97% of the sites with an llms.txt got zero requests for the file across the entire month. Search Engine Journal covered the same dataset.
The 3% that were fetched were mostly not fetched by the systems site owners had in mind. According to that Ahrefs study, named AI bots made up under a fifth of requests to those files, the heaviest readers were a training crawler and a coding agent rather than any answer engine, and the retrieval bots that answer live user queries accounted for 1.1% of requests. SEO audit tools checking whether the file existed were a larger source of traffic than AI search.
A controlled single-site test found the same. Otterly.ai watched AI bot traffic to its own llms.txt for 90 days and logged 84 requests out of 62,100 AI bot visits, which is 0.1% of the AI traffic the site received. The file was fetched less often than an average page on the same domain.
Correlation studies on the outcome side agree with the log studies on the input side. SE Ranking tested roughly 300,000 domains using correlation tests and a gradient-boosted model, and reported no significant relationship between publishing llms.txt and how often a domain was cited in major AI answers. Its crawl found the file on 10.13% of domains, spread fairly evenly across traffic tiers rather than concentrated among large brands.
| Study | Scale | What it measured | Finding |
|---|---|---|---|
| Ahrefs, 2026 | 137,210 domains, May 2026 | Server requests for the file | 97% received zero requests; AI retrieval bots were 1.1% of requests |
| SE Ranking, 2026 | About 300,000 domains | Correlation with citation frequency | No significant relationship; file present on 10.13% of domains |
| Otterly.ai, 2026 | One site, 90 days | AI bot requests to /llms.txt | 84 of 62,100 AI bot requests, or 0.1% |
| Originality.AI, 2026 | Over 3 million sites tracked | Adoption over time | 4,088 files in June 2025 to 36,120 in May 2026, growth of 8.8 times |
Have Google, OpenAI or Anthropic said they use it?
Google has said plainly that it does not. John Mueller of Google Search wrote on Bluesky on 17 June 2025 that "FWIW no AI system currently uses llms.txt", and Gary Illyes repeated the position at Search Central Live in July 2025, saying Google does not support the standard and has no plans to add it. Google has since suggested site owners noindex the file so it does not clutter search results.
The other model providers have said nothing that amounts to support. OpenAI, Anthropic and Meta have made no public commitment to read llms.txt in their production answer systems, and no provider documentation lists it as an input to retrieval. Absence of a statement is weaker evidence than Google's explicit denial, but it lines up with the crawler logs, which show the retrieval bots skipping the file even where it exists.
Worth separating from all of that: Anthropic, Stripe, Vercel and Cloudflare all publish an llms.txt for their own documentation. Publishing one is not the same as consuming one. Those companies ship the file because their users point coding assistants at their docs, which is the use case where a curated index does measurable work.
What no engine has claimed is the thing site owners most want to hear, which is that the file influences whether a page is cited. Until a provider says otherwise, treat llms.txt as a publishing convention with a real audience among developer tools and no demonstrated audience among answer engines. Our note on whether to block AI crawlers like GPTBot covers the file that does change what engines can do with your site.
Why are so many sites publishing one anyway?
Sites keep publishing llms.txt because the file is almost free to create and the promise is easy to believe, not because anyone has shown a return. Originality.AI, which tracks llms.txt across more than 3 million sites, recorded 4,088 files in June 2025 and 36,120 by May 2026, growth of about 8.8 times in twelve months. Its count including the companion llms-full.txt and ai.txt formats reached 38,980 sites.
Adoption at the top of the web is thinner than the growth rate suggests. Rankability's June 2026 count put 8.7% of the world's top 1,000 sites behind an llms.txt. It could not reach a verdict on 451 of those 1,000, most of them CDN, DNS and cloud API hosts that serve nothing at the root and so cannot publish the file at all. Among the 549 it could reach, the share was 15.8%.
The incentive structure explains the rest. A file that takes twenty minutes to write, carries no risk, and might help is an easy recommendation for an agency to make and an easy box for a client to tick. Nobody gets blamed for having one. The cost is not the file, it is the attention: a team that believes it has addressed AI visibility by publishing a text file has spent its AEO budget on the cheapest available placebo.
For the record, this site publishes an llms.txt. It is there because we would rather an assistant fetching our documentation read a clean summary than scrape the marketing pages, and because it costs nothing to maintain. We do not count it as a citation lever and we do not report it as work. Anyone selling it as one should be asked for the measurement.
What should you do instead of publishing llms.txt?
Instead of publishing llms.txt, spend the effort on the things the engines demonstrably read, starting with whether they can read you at all. Crawler access, server-rendered content, clean HTML and a working sitemap decide whether your pages enter a retrieval set in the first place, and a page that fails there cannot be rescued by a file listing it. Our guide to checking whether AI can read your website works through that in order.
Then work on the page level, because retrieval gets you considered and content gets you chosen. Answer each buyer question outright in the first sentence under its heading, keep sections able to stand alone for a reader who arrived mid-page, and give every number a named source in the same paragraph. Those are the changes the published research on generative engine optimisation actually measured, and they are set out in our AI citation checklist.
Add the part most sites skip, which is presence on the third-party pages engines retrieve for buying questions. A comparison article, a review profile or a community thread that names you is often what gets cited for "best option for X", and no amount of on-site tidying substitutes for it. Our methodology sets out how much weight the research puts on that cross-domain presence.
Keep llms.txt if you have one. Write it for developer tools, keep it accurate, and stop expecting it to move citations. Then measure the things that do, so that the next time a tactic arrives with a good story and no data, you already have a baseline to test it against.
What else do people ask about llms.txt?
Does llms.txt work for AI search visibility?
No, on every measurement published so far. Ahrefs analysed 137,210 domains and found that 97% of the ones publishing an llms.txt received zero requests for the file across the whole of May 2026. SE Ranking tested roughly 300,000 domains and found no relationship between having the file and how often a domain is cited in major AI answers. Publishing one is cheap and harmless, but nothing in the evidence connects it to being cited.
What is llms.txt supposed to do?
An llms.txt file is a plain markdown file at the root of a domain that lists the pages a site considers most useful to a language model, with a short description of each. The proposal treats it as a curated map for machines, in the way robots.txt is a set of crawl rules and sitemap.xml is a list of URLs. Nothing in the proposal binds any engine to read it, and that is the gap between the idea and the evidence.
Do AI crawlers actually request llms.txt?
Almost never. Ahrefs found that across the 3% of llms.txt files that did get any request in May 2026, the bots answering user queries in AI search accounted for just 1.1% of those requests, with SEO audit tools and training crawlers making up most of the traffic. Otterly.ai watched AI bot traffic to its own llms.txt for 90 days and recorded 84 requests out of 62,100 AI bot visits, or 0.1%.
Has Google said whether it uses llms.txt?
Yes, and the answer was no. Google's John Mueller wrote on Bluesky on 17 June 2025 that "FWIW no AI system currently uses llms.txt", and Gary Illyes told Search Central Live in July 2025 that Google does not support the standard and has no plans to. OpenAI, Anthropic and Meta have made no public commitment to read the file in their production answer systems either.
When is llms.txt worth publishing?
When your users point coding agents and assistants at your documentation. That is the one use case with real adoption: Stripe, Vercel, Cloudflare, Anthropic and most modern API products ship an llms.txt because developers ask an assistant to read their docs, and a curated index saves the agent from guessing. If your goal is being cited when a buyer asks an answer engine about your category, the file is not the lever.