llms.txt: What It Actually Does, and Why It Rots

When ChatGPT, Claude or Perplexity answers a question about your product, it is not consulting a decade of PageRank. It fetches a handful of pages and tries to work out what your site is. That is a very different retrieval problem from classic search, and most sites are accidentally hostile to it.

Why sitemaps are the wrong mental model

A sitemap optimises for coverage — every URL, every paginated archive, every tag page. That is correct for a crawler with a huge budget and a ranking model to sort the noise afterwards.

An LLM landing on your site has neither. It has a limited context window and one shot. If the first thing it ingests is 400 URLs of ?page=17 and /tag/misc, your three genuinely useful guides are buried.

llms.txt inverts this. It is a small markdown file at your root that optimises for priority:

# Your Product



> One-line description of what this actually does.



## Docs

- [Quickstart](https://example.com/docs/quickstart): Install and first request in 5 minutes

- [API Reference](https://example.com/docs/api): Every endpoint with request/response examples

Two rules make or break it:

1. 
   Every link carries a description. The colon-suffix annotation is what lets a model decide whether to fetch a page. A bare link list is barely better than a sitemap.


2. 
   Omit aggressively. If a page does not answer a question someone would ask, it does not belong.

llms-full.txt and the context tradeoff

llms-full.txt inlines expanded content rather than linking out, so a model can ingest everything in one request. This is genuinely useful for compact docs — and actively harmful for large sites, where you will blow the context window and get truncated mid-document.

Rough heuristic: if your docs exceed roughly 50k tokens, ship llms.txt alone and let models fetch selectively.

The part nobody mentions: it rots

This is where most implementations quietly fail. You write the file, ship it, and three months later half the descriptions describe features you renamed and two links 404. Nothing in your build catches it, because nothing validates it.

Two things worth automating:

• 
  Link liveness — CI check that every URL in llms.txt still returns 200


• 
  Regeneration on docs change — treat it as a build artifact, not a hand-maintained file

I ended up building a free generator for exactly this loop, after my own docs got cited with a description I had rewritten months earlier. Paste a URL, it crawls the structure, and produces spec-compliant llms.txt and llms-full.txt you can edit before exporting — reorder, drop low-value pages, rewrite descriptions. No signup, nothing stored: aicrawlable.com.

Worth being honest about

No search engine has committed to llms.txt as a ranking signal, and it may never become one. What it does today is cheap insurance: when a model does fetch your site, it reads a curated map instead of guessing. That asymmetry — near-zero cost, meaningful upside — is the whole argument.