llms.txt is a proposal, not a standard. Should you ship one?

Jeremy Howard proposed llms.txt in September 2024 as a website convention: a markdown file at the root telling LLMs what the site is about, with curated links to the most important content. Think robots.txt meets a site-map meets an elevator pitch. The question in every marketing channel since has been "should we ship one."

What the spec says

Two files: /llms.txt (curated summary and links) and /llms-full.txt (full markdown content of the site for ingestion). The proposal is explicitly for the LLM context window, not for crawling or ranking. It is not a signal any major LLM vendor has publicly committed to using.

Who is actually respecting it

As of October 2024: nobody officially. Anthropic, OpenAI, Google, Meta. None have said they read llms.txt. A handful of smaller AI tools (Perplexity via community patches, some coding assistants) read it opportunistically. The reference implementation has ~1,800 GitHub stars.

Why the hype is ahead of the substance

We have seen three vendors pitch llms.txt as a paid deliverable in 2024. The value proposition: your site gets cited more often by LLMs. There is no evidence for this. The LLMs that drive the vast majority of answer traffic (ChatGPT, Perplexity, Claude, Gemini) do not document reading llms.txt. Claims to the contrary are unfalsifiable until the vendors publish their ingestion specs, which they do not.

The case for shipping one anyway

• It costs roughly 2 hours of work to generate from an existing sitemap.

• If a standard emerges, early adoption costs you nothing.

• The exercise of writing a curated "what is this site" in markdown is genuinely useful for your own team.

The case against making it a priority

• It is competing for attention with things that demonstrably work (structured data, entity clarity, citation-worthy content).

• It has no measurable outcome today. You cannot A/B test it.

• Agencies are selling it as an AI-SEO service. Be skeptical of anyone charging more than a few hundred dollars for a markdown file.

Our call

Ship one if you can automate it from a CMS in an afternoon. Do not pay anyone to write one. Do not expect measurable impact. Revisit in 12 months.

Sources: llmstxt.org; GitHub repo activity, Oct 2024; community threads on Hacker News and r/TechSEO.