llms.txt

llms.txt is a proposed standard file, placed at the root of a website (e.g. /llms.txt), that gives large language models a curated, markdown-formatted summary of a site's most important content. Unlike robots.txt, which tells crawlers what they're allowed to access, llms.txt tells AI systems what the site actually contains and where to find the most relevant pages, in a format optimized for a model's limited context window.

What Is llms.txt?

Websites are built for human browsers: navigation menus, JavaScript rendering, ads, and visual layout. This works poorly for LLMs, which need clean, dense, plain-text information and have limited space to process it. llms.txt addresses this by giving site owners a way to hand LLMs a short, structured index instead of forcing them to parse full HTML pages.

The file lives at the root of the domain, follows a defined markdown structure, and typically links out to the site's key pages, often with a one-line description of each. It is a proposal introduced in 2024, not yet a formally ratified web standard, but adoption is growing among developer tools, documentation sites, and AI-focused companies.

llms.txt File Format

A valid llms.txt file follows a specific markdown structure:

# Project Name

> A one or two sentence summary of what this site or project is about.

Optional paragraph with additional context that an LLM should know before reading further.

## Docs

- [Getting Started](https://example.com/docs/getting-started): Setup and installation guide
- [API Reference](https://example.com/docs/api): Full endpoint documentation

## Examples

- [Quickstart Example](https://example.com/examples/quickstart): Minimal working example

## Optional

- [Changelog](https://example.com/changelog): Version history

llms-full.txt

Many sites that publish llms.txt also publish a companion file, llms-full.txt, at the same root path. Instead of just linking out to pages, llms-full.txt inlines the entire content of those pages into a single markdown document. This trades a larger file size for letting an LLM ingest everything in one fetch, without needing to follow individual links.

llms.txt vs robots.txt vs sitemap.xml

The three files are complementary, not competing. robots.txt controls access, sitemap.xml provides exhaustive coverage for search indexing, and llms.txt provides a hand-picked, LLM-readable summary of what matters most.