llms.txt explained — a markdown TOC for LLMs. What it does, why it exists, how to ship one this afternoon, and where the convention is heading.

• What it is A single markdown file at the root of your domain that tells AI systems which of your pages matter most, in what order, and how to think about them.
• What it is not Not a sitemap (those list every URL). Not a robots.txt (those allow/deny crawlers). A curated, opinionated guide written for an LLM that has a small attention budget for your site.
• Why now Shipping one is a one-afternoon job. Almost no one has done it yet — early adopters get disproportionate citation lift.

Search crawlers walk every URL they can find. AI training crawlers have a much smaller attention budget — the model only retains what it found contextually useful, and operators are increasingly selective about what makes the next training cut. Even on-demand retrieval bots (ChatGPT-User, Claude-Web, PerplexityBot) have to decide which pages on your domain to fetch in the few seconds they have to answer a user.

• What it solves A curated, hand-written file that says, in plain words: start here, in this order, with this framing. Compresses the "most important pages on this domain" signal into something an LLM can read in a single fetch.
• Origin Proposed by Jeremy Howard (Answer.AI) in 2024. By mid-2026 adopted by GitHub, Anthropic, Cloudflare, Stripe, Vercel and a long tail of developer-focused companies.
• Standardisation status IETF hasn't standardised it yet, but the format has stabilised enough that major LLM operators look for it in practice.

Three files at the root of your domain, three different jobs. The clearest way to think about them:

• robots.txt Who is allowed to fetch what. Permission. Binary.
• sitemap.xml Here are all my URLs. Completeness. Machine-readable.
• llms.txt Here is what matters, and how to think about it. Editorial. Human-readable.
• They layer, not compete robots.txt grants access, sitemap.xml exposes inventory, llms.txt exposes priority. An AI system reading all three of yours will end up with a much more accurate model of your business than one reading any single file in isolation.

Lives at /llms.txt on your domain. Should be served with content type text/markdown — most CDNs default it to text/plain, which works but is less correct. Shape is loose but converges on:

• H1 + tagline Single # H1 with the project or company name, followed by a one-line summary.
• Blockquote framing A short blockquote (>) framing the project — what it is, who it is for, what makes it distinctive. Two to three sentences max.
• Section headers (H2) Group pages by purpose. Common sections: Docs, API, Tools, Examples, Reference, Community, Company.
• Bulleted links Each section is a bulleted list. Each link gets a short descriptor after a colon — one sentence on what the page is and when it matters.
• Optional section An ## Optional section at the end with secondary pages LLMs can skip if they are short on context budget.

What a good llms.txt looks like for a SaaS company. Every line pulls weight — no filler, no marketing prose, every link has a clear job:

# DomainScan

> The diagnostic layer for every domain on the internet.
> 50+ tools, one stable JSON contract, free for casual use.
> Used by security teams, SREs, and brand-protection operators.

## Docs
- [Quickstart](https://domainscan.in/docs/quickstart): 5-minute setup, first API call
- [Authentication](https://domainscan.in/docs/auth): API keys and rate limits
- [Errors](https://domainscan.in/docs/errors): every error shape, with examples

## Tools
- [Domain Lookup](https://domainscan.in/domain/lookup): WHOIS + RDAP merged
- [Domain Trust Score](https://domainscan.in/domain/trust): all-in-one composite trust rating
- [AI Readiness](https://domainscan.in/domain/ai-ready): 5-signal audit
- [SSL Info](https://domainscan.in/security/ssl-info): certificate chain inspection

## Guides
- [Verify a website before paying](https://domainscan.in/blog/verify-website-before-paying): 7-step playbook
- [Payment scam red flags](https://domainscan.in/blog/payment-scam-red-flags): 12 patterns to watch for
- [AI readiness in 2026](https://domainscan.in/blog/ai-readiness-2026): the five signals that move citations

## Company
- [About](https://domainscan.in/about): team, mission, history

## Optional
- [Pricing](https://domainscan.in/pricing): paid tiers (free is enough for most users)
- [Privacy](https://domainscan.in/privacy): data handling policy

• One rule Write it for a smart-but-busy reader who has 30 seconds. If a section doesn't earn its bullet, cut it. The instinct to be comprehensive is the wrong instinct here — sitemap.xml is for completeness, llms.txt is for selection.

Community is still figuring out conventions. Most-cited examples in mid-2026 — each takes a slightly different stance on how much narrative belongs in the file versus the linked pages. Read all three before writing yours:

• Anthropic (anthropic.com/llms.txt) Tight, almost terse. Heavy use of blockquote framing. Sections for Products, Docs, Research. A model for B2B SaaS.
• GitHub (github.com/llms.txt) More verbose, with longer descriptors per link. Targets the "explain GitHub to someone who has never used it" case.
• Cloudflare (cloudflare.com/llms.txt) Heavy on docs and reference, light on marketing. A useful model for any company whose product is fundamentally technical.
• Stripe (stripe.com/llms.txt) Uses the optional section heavily, pushing secondary content (changelogs, blog) below the main lineup.
• Vercel (vercel.com/llms.txt) Pairs llms.txt with a separate llms-full.txt that includes the actual page contents inline. A two-file pattern gaining traction.

Four steps, each under 30 minutes:

• 1. Pick your top 8–12 URLs Not all of them — the most important ones. The pages you'd want someone reading if they only had 5 minutes to understand your business.
• 2. Write one sentence per URL What is this page, and when should an LLM use it? Resist the urge to write marketing copy. Be plain.
• 3. Group them Docs, Tools, Guides, Company — whatever fits. Three to five sections is the sweet spot. Fewer than three is too thin; more than five is harder to read.
• 4. Serve it Drop the file at /llms.txt in your public directory. Set content type to text/markdown if your CDN lets you. Add the URL to your sitemap.xml so it gets discovered. Done.

• Writing marketing copy instead of descriptors "The world's most powerful domain analytics platform" tells the LLM nothing. "WHOIS and RDAP merged, with domain age and registrar lookup" tells it everything.
• Listing every page If your llms.txt has 80 links, you've written a sitemap. Cut it to 12.
• Linking to login-gated pages The LLM can't read them. Link to public documentation, not the private dashboard.
• Forgetting the H1 and blockquote Without them, the LLM has no overall framing for your domain. The first 100 words of llms.txt do the heaviest lifting.
• Letting it go stale If you ship a new flagship product, update the file. We've seen sites where llms.txt still references a deprecated product as the headline.

Four directions the convention is moving in 2026:

• llms-full.txt Companion file that inlines the actual content of each linked page so an LLM can ingest the "essential" corpus of your site in a single fetch. Vercel and Anthropic both ship one. Expect this to become standard for documentation-heavy sites.
• Per-section access controls Some operators experimenting with marking sections as "allow training" vs "allow retrieval only" — a finer-grained version of robots.txt.
• Discovery via well-known URLs RFC-like .well-known/llms.txt path is gaining traction as an alternative location. Most operators ship both for now.
• Verification signals Cryptographic signing of llms.txt contents to prove they came from the domain owner, similar to how security.txt evolved. Early days, but the conversation has started.
• Headline trajectory llms.txt is becoming load-bearing infrastructure for how AI systems read the web. Teams shipping a thoughtful one now are buying themselves a year of head start before it becomes table stakes.

• What is llms.txt? A markdown file at the root of your domain that tells AI systems which of your pages matter most, in what order, and how to think about them. Editorial — you choose what goes in — and acts as a curated table of contents for LLMs.
• Where does llms.txt go? At /llms.txt on your domain root. Some sites also ship it at /.well-known/llms.txt. Serve with content type text/markdown when possible.
• Is llms.txt required? No. No enforcement, no penalty for not having one. But in mid-2026, having a well-written llms.txt is one of the highest-leverage AI-readiness moves you can make, simply because so few sites have shipped one yet.
• Does Google use llms.txt? Not for classical search ranking. Whether Google AI Overviews or Gemini use it is undocumented, but the major OpenAI, Anthropic and Perplexity bots are known to factor it in. Cost-to-benefit ratio favors shipping it regardless.
• How long should llms.txt be? Short. The best examples in the wild are 30–80 lines. Anything over 200 lines suggests you've slipped into sitemap territory. If you need to expose more, ship an llms-full.txt companion file rather than bloating the main one.
• Does llms.txt replace sitemap.xml? No. Different jobs. sitemap.xml tells crawlers about every URL — still ship one. llms.txt tells AI systems which URLs matter most. Together they cover inventory and priority.