Why llms.txt Alone Won’t Make AI Recommend Your Brand

What llms.txt is — and what it promised

llms.txt is a proposed standard published in September 2024 by Jeremy Howard, co-founder of Answer.AI. The idea: a Markdown file at the site root (/llms.txt) working as a curated index for AI systems — the brand's essential content, stripped of navigation, ads, and interface noise, easy to consume for models with limited context windows.

The original intent was reasonable and the problem it describes is real: modern HTML pages carry a lot of noise for a system that only wants the content. The proposal, however, carries a built-in premise rarely said out loud: it only works if AI systems decide to read the file and trust it. It is a unilateral proposal — the site declares; nobody committed to listening.

The comparison with successful standards is instructive. robots.txt has decades of adoption and a clear enforcement mechanism. Schema.org was born from a coalition of Google, Bing, Yahoo, and Yandex — the standard's consumers participated in its creation. llms.txt was born on the publishers' side, with no major consumer committed. That difference of origin explains almost everything that came next.

What the platforms say — in the documentation, not on stage

Mueller's parallel with the keywords meta tag deserves a paragraph. Search engines abandoned that tag because signals declared by the site itself, at zero cost and with no verification, are infinitely inflatable — and therefore useless for ranking trust. llms.txt repeats the structure of the problem: if every site can publish a flattering manifesto about itself, the manifesto differentiates no one. Modern answer systems select sources by signals the site does not control alone.

What the servers show: the file nobody reads

If official statements left any doubt, telemetry resolves it. Ahrefs analyzed real request behavior across a base of approximately 137,000 domains with llms.txt implemented. The findings:

• 97% of the files never received a single request. It's not that bots read and ignored them — they never fetched them.
• Of the requests that did exist, most came from SEO audit tools and non-AI crawlers — meaning the ecosystem that sells the tactic is its main consumer.
• Statistical models applied to the base found no effect of the file on citations in AI answers.

Meanwhile, the same server logs show what AI bots actually request: the site's normal HTML pages, for training and for answer grounding. The operational conclusion is direct — the systems already read your site; what decides the citation is not the file format, it is what they find when they read.

Where llms.txt has legitimate use — the part the hype leaves out

An honest article must register the other side: sophisticated tech companies — Stripe, Cloudflare, Vercel, Anthropic, among others — maintain llms.txt files. They are not naive. They are precise about the file's target audience: coding agents and agentic browsers that, in specific development workflows, choose to consume technical documentation in a clean format.

That is a real, narrow use case: API and technical product documentation, consumed by tools the developer points at the file. It has no relation to the scenario sold in the GEO market — that ChatGPT or AI Overviews would read the manifesto and start recommending the brand in commercial answers. Confusing the two scenarios is the central error of the discourse that overrates the tactic.

Why the tactic sold so well

If the evidence is so weak, why did llms.txt become a line item in so many GEO proposals? Because it has the three properties of the perfect tactic to sell — and none of the ones that make a tactic work:

• It is binary and screenshot-able. "We deployed the file" fits a checklist and a delivery screenshot. Semantic authority does not.
• It is cheap to execute and expensive to look like. Generating a Markdown file takes minutes; presenting it as "preparing the site for the AI era" sustains an invoice line.
• It cannot be refuted by the client. Since the promised effect is diffuse ("AIs will understand the site better"), the absence of results is never attributed to the file.

The pattern is not new. Every technology transition produces its layer of technical talismans — simple-to-implement objects sold as shortcuts to a result that actually depends on structural work. Recognizing the talisman is the first filter for evaluating any AI visibility vendor: ask what evidence supports each deliverable and what exactly it measures after delivery.

What actually predicts citation: the data

The right question was never "which file to put on the server" — it is "what do AI systems use to decide whom to cite". Here the evidence base is richer, and three independent studies, with different methodologies, converge:

Two method caveats, because rigor is part of the thesis. First, correlation is not causation — Ahrefs' own researchers stress that the data shows AI-visible brands also have broad web presence, not that manufacturing mentions mechanically generates citations. The underlying signal is brand authority as a composite, absorbed by the systems over time. Second, the AI measurement market is young and mostly operated by tool vendors; the numbers should be read as order of magnitude and direction, not physical constants.

Caveats made, the direction is unambiguous: all the strong signals live outside the brand's server. No independent study found, among the relevant predictors, any self-declared technical artifact.

The mechanism: how AI decides whom to cite

1. The question triggers a search

   The major answer systems ground their responses with classic web search. Whoever is strong in organic enters the candidate set; whoever isn't indexable doesn't even exist for this stage.

2. The system reads a small set of sources

   From the search, the model receives a handful of high-trust pages — frequently editorial outlets, communities, encyclopedias, reviews. This is where earned media weighs: the brand needs to be present inside the pages the system selects, not only on its own site.

3. The model synthesizes — and cites what repeats with consistency

   If the brand appears consistently described across multiple independent sources, the model treats it as the probable answer. If it appears rarely, or described in contradictory ways, it is diluted in the synthesis. Recurrence + consistency + source trust: that is the tripod the correlation studies capture.

Seen through this pipeline, llms.txt's limit becomes obvious: it acts at a point that does not participate in the decision. The file doesn't improve the brand's position in the search that grounds the answer, doesn't insert it into the third-party sources the model reads, and doesn't increase the consistency with which the ecosystem describes it. It is plumbing — in an architecture problem.

The strategic reading: semantic authority is an asset, the file is an accessory

For the decision-maker, the conclusion matters more than the technicality. AI visibility is not a technical compliance problem solved with a file checklist — it is an asset-building problem: making the brand the entity the digital ecosystem describes with clarity, consistency, and proof, in the sources the systems trust.

• Distrust GEO projects whose central deliverable is a self-declared technical artifact. If the proposal revolves around files, tags, and manifestos, it is optimizing the point of the pipeline that decides nothing.
• Demand the authority layer in scope. Qualified mentions, editorial presence, publishable proprietary data, visible experts, entity consistency across channels — that is where the studies locate the effect.
• Require measurement of what matters. Not "file deployed," but brand presence in real answers, per platform, compared against competitors, over time.

What to do instead, in order

1. Secure the foundation the systems actually read. Crawlability, indexing, performance, and coherent structured data on the HTML pages — the true input of grounding.
2. Structure content as answers. Real market questions, direct answer up front, depth and proof after it — the format that survives synthesis.
3. Consolidate entity identity. The brand described the same way — what it does, for whom, with what differentiator — on the site, in the press, in reviews, directories, and profiles. Inconsistency dilutes citation.
4. Build mentions where the systems read. Editorial presence, proprietary data that earns coverage, qualified participation in communities and review platforms relevant to the category.
5. Measure citations continuously. Search Console (generative AI experiences), Bing Webmaster Tools (Copilot), and structured prompt tests with declared methodology.
6. Only then, if you want, publish llms.txt. At the end, for completeness and at zero marginal cost — never as the center of the project.

Frequently asked questions