What Is llms.txt? The Spec, Examples and Honest Verdict

What is llms.txt? Originally proposed by Jeremy Howard at Answer.AI, llms.txt is a standardized Markdown file placed at a website's root that gives AI models a curated index of key documentation and clean text endpoints. When autonomous agents explore the modern web, parsing raw HTML navigation bars, tracking scripts, cookie banners, and client-side JavaScript bundles consumes excessive context window tokens. The proposed standard solves this by providing a clean, token-efficient roadmap built specifically for machine ingestion.

What is llms.txt? The proposed standard

At its core, an llms txt file is a standardized Markdown document placed at /llms.txt on your domain root. It provides a human-readable and machine-friendly directory of your site's most critical pages, summarized concisely to maximize token efficiency.

https://yourdomain.com/llms.txt

Unlike traditional web feeds, llms.txt does not rely on complex XML tags or verbose JSON schemas. Instead, it uses clean Markdown formatting (# H1, blockquotes, and unordered list items), making it trivial for any LLM to parse using standard text splitters.

We practice this directly at MrScraper: you can inspect our own mrscraper.com/llms.txt to see how we structure endpoint summaries and SDK references for AI assistants.

The exact llms.txt spec format: How to write one

The official llms.txt spec (v2 proposal maintained at llmstxt.org) defines a clean structural layout. The file must be formatted strictly in Markdown and follow four layout rules:

1. H1 Title: The file MUST start with a single # H1 heading containing the project or site name.
2. Blockquote Summary: A short blockquote (> Summary) immediately following the H1, explaining what the site or project does in 2–4 sentences.
3. Optional Context Paragraphs: Markdown prose providing background guidance for the AI reader.
4. Sectioned Link Lists: H2 headers (## Section Name) grouping markdown bullet links with short descriptions formatted as - [Page Title](url): Concise summary.

# Project Name

> Project Name provides high-performance web APIs for developer automation. This file outlines key documentation for AI assistants and automated agents.

Optional background context explaining system constraints or architectural defaults.

## Core Documentation

- [API Reference](https://example.com/docs/api): Complete OpenAPI specs and endpoint documentation.
- [Authentication](https://example.com/docs/auth): Guide on Bearer tokens and API key rotation.

## Optional Resources

- [Architecture Guide](https://example.com/docs/arch.md): High-level system topology diagram.

What is llms-full.txt?

The specification also introduces llms-full.txt. While llms.txt acts as an index of linked URLs, llms-full.txt concatenates the complete text of all essential documentation into a single contiguous Markdown file. This allows AI systems with large context windows (such as Gemini 1.5 Pro or Claude 3.7 Sonnet) to ingest your entire documentation suite in a single HTTP request without executing recursive crawler loops.

Three llms.txt examples

Examining a real llms.txt example helps separate production-ready implementations from poor configurations.

1. Minimal llms.txt (Fast API Docs Index)

# Acme API

> High-throughput payment processing API infrastructure.

## Documentation

- [Getting Started](https://acme.dev/docs/start): Quickstart guide.
- [API Spec](https://acme.dev/docs/spec): Endpoints reference.

Commentary: Ideal for small microservices or open-source libraries. It uses minimal tokens while giving AI models exact entry points.

2. Complete Production-Grade llms.txt (Well-Curated)

# Cloud Platform Docs

> Scalable cloud infrastructure hosting for serverless functions, static assets, and edge databases.

This document guides AI agents to primary technical specifications.

## Essential Guides

- [Deployment Quickstart](https://cloud.example/docs/deploy): Step-by-step CLI deployment commands.
- [Environment Variables](https://cloud.example/docs/env): Managing secrets across staging and production.

## System Limits & Pricing

- [Rate Limits](https://cloud.example/docs/limits): Request quotas and concurrent execution boundaries.
- [Pricing Model](https://cloud.example/docs/pricing): Token and bandwidth billing calculations.

## Optional Full Context

- [Full Text Index](https://cloud.example/llms-full.txt): Complete concatenated documentation file.

Commentary: Excellent implementation. It provides clear sectioning, concise page summaries, and links to llms-full.txt for deep context models.

3. Bloated Anti-Pattern (Keyword Stuffed)

# Best Cheap Cloud API Provider Free Fast Tools 2026

> The best affordable cloud API provider with top performance and high availability.

## Links

- [Link 1](https://example.com/page1)
- [Link 2](https://example.com/page2)
- [Link 3](https://example.com/page3)
... (500 uncurated URLs without descriptions)

Commentary: Fails the spec. It lacks descriptive summaries, includes raw uncurated link dumps, and uses spammy keyword stuffing that wastes model context tokens.

llms.txt vs robots.txt vs sitemap.xml

A common area of confusion is how llms.txt relates to existing root-level web standards. Understanding llms.txt vs robots.txt requires evaluating target consumers and architectural intent.

While robots.txt controls access permissions and sitemap.xml lists URL structures, llms.txt delivers curated, token-efficient Markdown content directly to AI models.

Does llms.txt actually work in 2026?

The central question for engineering teams is straightforward: does llms.txt work in production today?

As of September 2026, adoption across the AI ecosystem falls into three distinct categories:

1. Active Adopters (Confirmed Support — Verified September 2026)

• Developer Documentation Platforms: Services like Mintlify, ReadMe, and GitBook automatically generate and serve llms.txt files for hosted docs out of the box.
• AI Search Engines & Agents: Perplexity, Claude Web Search agents, and custom GPT actions actively look for /llms.txt when exploring documentation domains to construct accurate citations.
• Edge Networks: Cloudflare provides edge workers to convert HTML sites into llms.txt indexes automatically.

2. Passive Consumers (Secondary Crawling — Verified September 2026)

• GPTBot & ClaudeBot: Primary AI web crawlers fetch llms.txt when present, using it to prioritize discovery of core documentation URLs.

3. Non-Adopters (Standard Indexing — Verified September 2026)

• Traditional Search Engines: Standard Google Search indexers continue to rely on HTML parsing, sitemap.xml, and structured schema.org data.

The Chrome Lighthouse Agentic Signal

A notable milestone in mid-2026 was Chrome Lighthouse introducing an experimental audit signal for agentic web readiness. Lighthouse checks for the presence of /llms.txt under its machine readability audit suite, signaling that browser vendors anticipate agentic web navigation becoming a standard pattern.

Which AI systems actually fetch llms.txt?

To provide concrete transparency, we tested and verified how major AI systems and crawlers interact with llms.txt in production environments:

How AI crawlers read sites without llms.txt

When a site lacks an llms.txt file, AI models and web crawlers must inspect raw HTML pages individually. They execute HTTP requests, strip boilerplate markup, extract text node content, and attempt to deduce heading hierarchy automatically.

For dynamic Single Page Applications (SPAs) built on modern JavaScript frameworks, AI crawlers face significant obstacles. Most AI crawlers still read pages the hard way, and many cannot run JavaScript at all. MrScraper's Web Scraper API returns clean structured content from any page, which is the same problem llms.txt is trying to solve from the other direction.

For a deeper look into web crawling mechanics, read our guides on JavaScript crawling for web scrapers, agentic search trends, the rise of agentic AI, web crawling vs web scraping, and our empirical benchmark comparing codex vs claude code on consuming live technical documentation.

The criticism and risks: Summary substitution & token manipulation

While llms.txt offers clear benefits, the proposal has drawn valid criticism from web standards developers and security researchers:

1. Summary Substitution Risk

Critics argue that by providing pre-written blockquote summaries in llms.txt, site owners risk AI search engines citing the summary instead of reading the source page. If the summary lacks nuance, the AI model may return superficial answers.

2. Generative Engine Optimization (GEO) Manipulation

Like meta keywords in early search engine history, unverified llms.txt files can be manipulated with biased claims. AI models must balance trusting curated site summaries against verifying facts in full source text.

3. Maintenance Staleness

If an engineering team updates API documentation but forgets to update llms.txt, AI assistants will direct users to outdated URLs or deprecated parameters, creating stale context.

How to generate an llms.txt file

Setting up your file takes only a few minutes. You can build it manually or use an automated llms.txt generator tool.

# Example CLI generation using node
npx llmstxt-generator --input ./docs --output ./public/llms.txt

Setup Steps:

1. Create the File: Create a plain Markdown file named llms.txt.
2. Add Header Metadata: Include your project H1 title and blockquote summary.
3. Curate Core Links: Group primary documentation URLs under H2 section headings with 1-sentence descriptions.
4. Deploy to Root: Upload the file to your root web directory (https://yourdomain.com/llms.txt).
5. Verify Access: Ensure your web server returns the text/markdown or text/plain Content-Type header without authentication blocks.

Frequently Asked Questions

Is llms.txt actually used by AI models?

Platforms like Mintlify, ReadMe, Perplexity, and Claude Web Search actively fetch and process llms.txt files as of September 2026. These systems use the file to discover canonical documentation URLs and construct accurate technical citations. Traditional search crawlers like Googlebot do not require llms.txt, relying instead on standard HTML rendering and XML sitemaps.

Is llms.txt worth it?

Creating an llms.txt file requires under twenty minutes and provides an immediate, token-efficient Markdown map of your documentation for AI systems. The file prevents model hallucinations by providing explicit page summaries and direct Markdown endpoints. Engineering teams maintaining developer documentation gain clearer AI citations with almost zero recurring maintenance overhead.

Is llms.txt mandatory?

llms.txt is a proposed community standard, not an official W3C or IETF web requirement. Websites function and index normally across all search engines without the file. Adding llms.txt remains an optional enhancement specifically designed to streamline content discovery for generative AI models and autonomous developer agents.

What is an llms.txt file?

An llms.txt file is a standardized Markdown document located at the root of a domain at /llms.txt. The file provides a human-readable and machine-friendly directory of a website's primary documentation and resources. AI models parse llms.txt to navigate technical documentation without wasting tokens on HTML boilerplate or navigation code.

What is the difference between llms.txt and robots.txt?

robots.txt controls crawler access permissions using directive rules like Allow and Disallow, while llms.txt provides curated content pathways for AI models. robots.txt instructs crawlers where they may go, but does not provide content. In contrast, llms.txt delivers clean, pre-summarized Markdown text specifically optimized for large language model context windows.

Does llms.txt help SEO?

llms.txt does not improve traditional Google organic keyword rankings, but it significantly enhances Generative Engine Optimization (GEO) and Answer Engine Optimization (AEO). AI assistants like Perplexity and ChatGPT prioritize curated Markdown summaries when generating answers. Having llms.txt increases the likelihood of accurate technical citations in AI-generated search summaries.

What is llms-full.txt?

llms-full.txt is an optional companion file that concatenates the complete text of all essential site documentation into a single contiguous Markdown document. While llms.txt functions as a curated link directory, llms-full.txt allows large-context models like Claude 3.7 Sonnet or Gemini 1.5 Pro to ingest an entire knowledge base in one HTTP request.

Where do I put the llms.txt file?

Place the llms.txt file in the root public directory of your web server so it resolves at https://yourdomain.com/llms.txt. Your server must serve the file over HTTPS with a text/markdown or text/plain Content-Type header. The file must remain publicly accessible without requiring user authentication, cookies, or redirect loops.

How do I generate an llms.txt file?

Generate an llms.txt file manually using standard Markdown syntax, or automatically using dedicated CLI generator tools like llmstxt-generator. Modern documentation frameworks such as Mintlify and Docusaurus generate llms.txt natively during static site builds. Edge platforms like Cloudflare Workers can also dynamically aggregate HTML pages into an llms.txt endpoint.

Do I need llms.txt if I already have a sitemap?

You should consider adding llms.txt even if you maintain an XML sitemap because the two files serve entirely different purposes. An XML sitemap lists raw page URLs for search indexers without context. An llms.txt file provides curated Markdown descriptions and contextual guidance that AI reasoning models can ingest directly.

Building an AI-ready web content architecture

Adopt llms.txt as a low-cost, high-efficiency Markdown index for AI models, while ensuring your core web pages remain accessible to automated scrapers and crawlers alike.

• Deploy a curated llms.txt file at your domain root to guide AI agents and technical search engines.
• Maintain concise, factual summaries across all linked documentation files to avoid summary substitution errors.
• Expose clean structured endpoints alongside your HTML pages so both human users and autonomous agents can consume content reliably.