llms.txt explained — a markdown TOC for LLMs

llms.txt is to AI crawlers what sitemap.xml is to search crawlers — except it’s markdown, it’s editorial, and it’s still finding its shape. Here is what it does, why it exists, how to ship one without overthinking it, and where the convention is heading in 2026.

TL;DR

• llms.txt is a single markdown file at the root of your domain that tells AI systems which of your pages matter most, in what order, and how to think about them.
• It is not a sitemap (those list every URL) and it is not a robots.txt (those allow/deny crawlers). It’s a curated, opinionated guide written for an LLM that has a small attention budget for your site.
• Shipping one is a one-afternoon job. Almost no one has done it yet, so the early adopters get disproportionate citation lift.

Why llms.txt exists

Search crawlers walk every URL they can find. AI training crawlers do the same, but their attention budget is much smaller — the model only retains what it found contextually useful, and the operators are increasingly selective about what makes it into the next training cut. Even on-demand retrieval bots (ChatGPT-User, Claude-Web, PerplexityBot) have to decide which pages on your domain to fetch in the few seconds they have to answer a user.

llms.txt is a curated, hand-written file that solves this. It says, in plain words: start here, in this order, with this framing. It compresses the “most important pages on this domain” signal into something an LLM can read in a single fetch.

The convention was proposed by Jeremy Howard (Answer.AI) in 2024. By mid-2026 it has been adopted by GitHub, Anthropic, Cloudflare, Stripe, Vercel, and a long tail of developer-focused companies. The IETF hasn’t standardized it yet, but the format has stabilized enough that the major LLM operators look for it in practice.

llms.txt vs sitemap.xml vs robots.txt

Three files at the root of your domain, three different jobs. The clearest way to think about them:

• robots.txt — who is allowed to fetch what. Permission. Binary.
• sitemap.xml — here are all my URLs. Completeness. Machine-readable.
• llms.txt — here is what matters, and how to think about it. Editorial. Human-readable.

You want all three. They don’t compete; they layer. robots.txt grants access, sitemap.xml exposes inventory, llms.txt exposes priority. An AI system reading all three of yours will end up with a much more accurate model of your business than one reading any single file in isolation.

The format, in detail

The file lives at /llms.txt on your domain and should be served with the content type text/markdown. Most CDNs default it to text/plain, which works but is less correct.

The expected shape is loose, but converges on:

• An # H1 with the project or company name, followed by a one-line summary.
• A short blockquote (>) framing the project — what it is, who it’s for, what makes it distinctive. Two to three sentences max.
• Section headers (##) grouping pages by purpose. Common sections: Docs, API, Tools, Examples, Reference, Community, Company.
• Each section is a bulleted list of links. Each link gets a short descriptor after a colon — one sentence on what the page is and when it matters.
• Optionally, an ## Optional section at the end with secondary pages that LLMs can skip if they’re short on context budget.

A full annotated example

Here is what a good llms.txt looks like for a SaaS company. Notice that every line pulls weight — there’s no filler, no marketing prose, and every link has a clear job.

# DomainScan

> The diagnostic layer for every domain on the internet.
> 50+ tools, one stable JSON contract, free for casual use.
> Used by security teams, SREs, and brand-protection operators.

## Docs
- [Quickstart](https://domainscan.in/docs/quickstart): 5-minute setup, first API call
- [Authentication](https://domainscan.in/docs/auth): API keys and rate limits
- [Errors](https://domainscan.in/docs/errors): every error shape, with examples

## Tools
- [Domain Lookup](https://domainscan.in/domain/lookup): WHOIS + RDAP merged
- [Domain Trust Score](https://domainscan.in/domain/trust): all-in-one composite trust rating
- [AI Readiness](https://domainscan.in/domain/ai-ready): 5-signal audit
- [SSL Info](https://domainscan.in/security/ssl-info): certificate chain inspection

## Guides
- [Verify a website before paying](https://domainscan.in/blog/verify-website-before-paying): 7-step playbook
- [Payment scam red flags](https://domainscan.in/blog/payment-scam-red-flags): 12 patterns to watch for
- [AI readiness in 2026](https://domainscan.in/blog/ai-readiness-2026): the five signals that move citations

## Company
- [About](https://domainscan.in/about): team, mission, history

## Optional
- [Pricing](https://domainscan.in/pricing): paid tiers (free is enough for most users)
- [Privacy](https://domainscan.in/privacy): data handling policy

Real-world examples in the wild

The community is still figuring out conventions. The most-cited examples in mid-2026 are below — each takes a slightly different stance on how much narrative belongs in the file versus the linked pages. Read all three before writing yours.

• Anthropic (anthropic.com/llms.txt) — tight, almost terse. Heavy use of the blockquote framing. Sections for Products, Docs, Research. A model for B2B SaaS.
• GitHub (github.com/llms.txt) — more verbose, with longer descriptors per link. Targets the “explain GitHub to someone who’s never used it” case.
• Cloudflare (cloudflare.com/llms.txt) — heavy on docs and reference, light on marketing. A useful model for any company whose product is fundamentally technical.
• Stripe (stripe.com/llms.txt) — uses the optional section heavily, pushing secondary content (changelogs, blog) below the main lineup.
• Vercel (vercel.com/llms.txt) — pairs llms.txt with a separate llms-full.txt that includes the actual page contents inline. A two-file pattern that’s gaining traction.

How to ship one this afternoon

The job breaks into four steps, each under 30 minutes:

1. Pick your top 8–12 URLs. Not all of them — the most important ones. The pages you’d want someone reading if they only had 5 minutes to understand your business.
2. Write one sentence per URL. What is this page, and when should an LLM use it? Resist the urge to write marketing copy. Be plain.
3. Group them. Docs, Tools, Guides, Company — whatever fits. Three to five sections is the sweet spot. Fewer than three is too thin; more than five is harder to read.
4. Serve it. Drop the file at /llms.txt in your public directory. Set the content type to text/markdown if your CDN lets you. Add the URL to your sitemap.xml so it gets discovered. Done.

Common mistakes to avoid

• Writing marketing copy instead of descriptors. “The world’s most powerful domain analytics platform” tells the LLM nothing. “WHOIS and RDAP merged, with domain age and registrar lookup” tells it everything.
• Listing every page. If your llms.txt has 80 links, you’ve written a sitemap. Cut it to 12.
• Linking to login-gated pages. The LLM can’t read them. Link to the public documentation, not the private dashboard.
• Forgetting the H1 and blockquote. Without them, the LLM has no overall framing for your domain. The first 100 words of llms.txt do the heaviest lifting.
• Letting it go stale. If you ship a new flagship product, update the file. We’ve seen sites where the llms.txt still references a deprecated product as the headline.

Where the standard is heading

A few directions the convention is moving in 2026:

• llms-full.txt. A companion file that inlines the actual content of each linked page, so an LLM can ingest the “essential” corpus of your site in a single fetch. Vercel and Anthropic both ship one. Expect this to become standard for documentation-heavy sites.
• Per-section access controls. Some operators are experimenting with marking certain sections as “allow training” vs “allow retrieval only” — a finer-grained version of robots.txt.
• Discovery via well-known URLs. The RFC-like .well-known/llms.txt path is gaining traction as an alternative location. Most operators ship both for now.
• Verification signals. Cryptographic signing of llms.txt contents to prove they came from the domain owner, similar to how security.txt evolved. Early days, but the conversation has started.

The headline trajectory: llms.txt is becoming load-bearing infrastructure for how AI systems read the web. The teams shipping a thoughtful one now are buying themselves a year of head start before it becomes table stakes. Our piece on AI readiness covers the broader signal lineup.

FAQ

What is llms.txt?

A markdown file at the root of your domain that tells AI systems which of your pages matter most, in what order, and how to think about them. It’s editorial — you choose what goes in — and acts as a curated table of contents for LLMs.

Where does llms.txt go?

At /llms.txt on your domain root. Some sites also ship it at /.well-known/llms.txt. Serve it with content type text/markdown when possible.

Is llms.txt required?

No. There is no enforcement and no penalty for not having one. But in mid-2026, having a well-written llms.txt is one of the highest-leverage AI-readiness moves you can make, simply because so few sites have shipped one yet.

Does Google use llms.txt?

Not for classical search ranking. Whether Google AI Overviews or Gemini use it is undocumented, but the major OpenAI, Anthropic, and Perplexity bots are known to factor it in. The cost-to-benefit ratio favors shipping it regardless.

How long should llms.txt be?

Short. The best examples in the wild are 30–80 lines. Anything over 200 lines suggests you’ve slipped into sitemap territory. If you need to expose more, ship an llms-full.txt companion file rather than bloating the main one.

Does llms.txt replace sitemap.xml?

No. They have different jobs. sitemap.xml tells crawlers about every URL; you should still ship one. llms.txt tells AI systems which URLs matter most. Together they cover inventory and priority.