Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

llms.txt: The Complete Guide (2026)

What llms.txt is, how to create one, how it differs from robots.txt and sitemap.xml, where it measurably helps (docs read by coding agents) and where it doesn't (AI search), and a rule for deciding by site type.

By Andrej Ruckij · · 5 min read

llms.txt: The Complete Guide (2026)

By Andrej Ruckij · June 16, 2026 · updated November 4, 2026

TL;DR: llms.txt is a proposed standard: a curated markdown file at your site root that gives AI systems a short map of your most important pages. It is a reading aid, not access control and not a ranking lever. Google’s Search docs say it “will neither harm nor help” your visibility, and no AI search crawler says it reads yours. The one place with measured results is documentation read by coding agents. My rule: on a docs, API or SDK site, ship it and keep it current; on a marketing site, store or blog, add it only if your CMS generates it for free.

What you’ll learn

  • What llms.txt is and the problem it tries to solve
  • How to create one, and what to put in it
  • How it differs from robots.txt and sitemap.xml
  • Who actually reads it, and whether you should bother

What llms.txt is

llms.txt is a plain markdown file you place at https://yoursite.com/llms.txt. It holds a curated map of your site: a title, a one-line summary, and grouped links to your most important pages, each with a short description. Jeremy Howard (Answer.AI) proposed it in September 2024, and the spec was revised as v2 in August 2026. The idea is to give large language models a short overview instead of making them infer your site’s structure from raw HTML. For the one-line version, see glossary/llms-txt.

The key word is curated. Unlike a sitemap, it doesn’t list every URL. It lists the handful that matter, with context.

How to create one

The format is minimal: an H1 (your brand), a blockquote summary, optional context prose, and H2 sections of links. Writing one by hand takes about 20 minutes. The full steps and a copy-paste template are in how-to-create-llms-txt. An optional companion file, llms-full.txt, holds the full text of your key pages for reference-heavy sites.

The most common mistake is dumping your entire URL list in. That gives you a worse-formatted sitemap, not an llms.txt.

How it differs from robots.txt and sitemap.xml

These three files get conflated all the time. They do different jobs:

  • robots.txt states access rules: which crawlers may fetch what. It’s a request. Compliant bots honour it, and a firewall is what actually stops the rest.
  • sitemap.xml helps discovery: an exhaustive list of URLs so search engines find your pages. It controls nothing.
  • llms.txt is a reading aid: a curated map of your best pages. It controls nothing either. Listing a page doesn’t override a robots.txt Disallow, and leaving a page out doesn’t hide it.

In a line: robots.txt is a permission sign, sitemap.xml is a complete index, and llms.txt is a guided tour. None replaces another.

Does it work?

For AI search, no evidence says so. In Ahrefs’ study of 137,210 sites, 97% of published files got zero requests in a month, and SE Ranking found no link between having the file and AI citations. For docs, yes: in Mintlify’s benchmark, a link to llms.txt cut the dead links Claude Code and Codex hit by about 90%. The log studies, vendor statements and benchmark are in does-llms-txt-work.

Where it fits in SEO and GEO

It won’t help or hurt Google rankings, including AI Overviews and AI Mode; Google says it will “neither harm nor help” your Search visibility. For getting cited in AI answers, it isn’t on the list of things that correlate with citations; that list is in seo/ai-visibility. More in does-llms-txt-work.

Is it worth it?

It depends on who reads your site:

  • Docs, API, SDK or dev tool: yes. Generate it from your docs build and keep it current. Coding agents use it.
  • Marketing site, store, blog, local business: only if your CMS generates it for free. Don’t hand-maintain one for AI search.

The money cost is close to nothing. The real cost is upkeep: a stale file sends agents to dead links, which is the exact problem the docs benchmark measured. If nothing regenerates it, a file you forget about is worse than none on a docs site.

Common questions

  • Q: Is llms.txt worth it? A: On a docs or API site, yes. On a marketing site, only if your platform does it for you. See does-llms-txt-work.
  • Q: Does llms.txt help me rank on Google? A: No. Google’s Search docs say it will neither harm nor help, including in AI Overviews and AI Mode.
  • Q: Is llms.txt the same as robots.txt? A: No. robots.txt states access rules; llms.txt is a map and controls nothing.
  • Q: Should I just convert my sitemap to llms.txt? A: No. Curation is the point; an exhaustive list defeats it.

Key takeaways

  • llms.txt is a curated markdown map of your best pages, at /llms.txt.
  • robots.txt states access rules (a request, enforced only by a firewall); sitemap.xml and llms.txt control nothing.
  • Google says it neither helps nor hurts Search, and no AI search crawler says it reads yours.
  • The measured benefit is on documentation read by coding agents.
  • Docs site: ship it and keep it current. Marketing site: only if your CMS makes it for free.

Sources

#llms-txt#geo#ai-crawlers#seo#ai-visibility