Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

Does llms.txt Actually Work? What the Server Logs Show in 2026

Which AI systems actually fetch llms.txt (Ahrefs, EZY and SE Ranking data), what Google, OpenAI, Anthropic and Perplexity say on their own pages, where the file measurably helps (docs read by coding agents), how to check your own logs, what it costs, how it fails, and a decision rule by site type.

By Andrej Ruckij · · 18 min read

Does llms.txt Actually Work? What the Server Logs Show in 2026

By Andrej Ruckij · June 16, 2026 · updated November 4, 2026

TL;DR: For getting cited by AI search, no. For documentation read by coding agents, yes, and that’s the only place with measured results. Google’s Search docs say an llms.txt “will neither harm nor help” your visibility. OpenAI, Anthropic and Perplexity publish llms.txt files for their own developer docs, but none of their crawler pages says their bots read yours. Server logs agree: in Ahrefs’ study of 137,210 sites, 97% of published files got zero requests in a month, and in EZY’s 12-week test on 83 sites OpenAI’s crawlers fetched llms.txt 7 times against 3,990 robots.txt fetches. SE Ranking found no link between having the file and AI citations across about 300,000 domains. The positive evidence is all on docs: in Mintlify’s benchmark, a link to llms.txt cut the dead links Claude Code and Codex hit by about 90%. My rule: docs site or API, ship it and keep it current; marketing site, store or blog, add it only if your CMS does it for free, and spend the real effort elsewhere.

This piece sits under our complete guide to llms.txt and answers the question the guides skip: does it do anything? For the file format, see how to create one.

The short version, by who’s reading

“Does it work” depends on which reader you mean. There are three kinds of software that might open your llms.txt, and the evidence is different for each:

ReaderExamplesDoes it fetch llms.txt?Does it use it?
Search and training crawlersGooglebot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBotRarely: GPTBot was 4.5% of llms.txt requests in Ahrefs’ data, OAI-SearchBot 0.74%No vendor says so; Google says it “will neither harm nor help”
Assistants fetching a page for a userChatGPT-User, Claude-User, Perplexity-UserRarely: all AI assistants together were 2.5% of requests in Ahrefs’ dataNo vendor says so
Coding agents and docs toolsClaude Code, Cursor, Codex, MCP docs serversYes, on docs sites; Claude Code was one of the most active AI fetchers in Ahrefs’ dataYes, measurably, in Mintlify’s benchmark

The original stance of this article was that llms.txt is cheap but unproven as a visibility lever. Everything published since June holds that up for AI search. What has changed is that the docs case now has real numbers behind it, so “unproven” undersells it for one type of site.

Who actually fetches llms.txt: the log studies

Ahrefs (June 15, 2026). Louise Linehan and Xibeijia Guan looked at 137,210 domains in Ahrefs Web Analytics that had traffic in May 2026. 28% published an llms.txt. Of those, 97% got zero requests that month. The remaining 3% (about 1,100 sites) received roughly 22,000 requests in total, 96% of them from bots.

Who those bots were is the interesting part:

  • SEO audit tools made the most requests, 21.7%. Ahrefs’ own crawlers alone were 10.6%.
  • All AI categories together were 19.5%: AI agents and agentic infrastructure 10.5%, AI training crawlers 5.3%, AI assistants 2.5%, AI retrieval bots (the ones that fetch pages for AI search answers) 1.1%.
  • GPTBot was the largest AI fetcher, at 4.51%. ClaudeBot was 0.8%, OAI-SearchBot 0.74%. PerplexityBot made fewer fetches than Slackbot, the link-preview bot.
  • Claude Code ranked second among AI fetchers, ahead of every retrieval bot, assistant and other training crawler.
  • Googlebot made about 900 fetches, which the authors put down to ordinary crawling.
  • When a site had no llms.txt, AI bots made zero requests for it. 98% of requests to missing files were human.

Ahrefs is upfront about the limits: its customers skew technical and SEO-minded, so 28% adoption is “an upper bound”, and fetching a file isn’t reading it. In their words: “Every figure in this study is therefore a ceiling on actual llms.txt consumption.”

EZY.ai (July 27, 2026). EZY put llms.txt on 83 small-business sites from its panel and logged verified crawlers for 12 weeks (April 27 to July 19, 2026):

Crawlerrobots.txt fetchesllms.txt fetches
OpenAI (GPTBot family)3,9907
Anthropic (ClaudeBot)3,1209
PerplexityBot7750
Googlebot5,12567
Meta-ExternalAgent172193

Their conclusion: “The one major AI company reading llms.txt at scale right now is Meta.” Two caveats they state themselves: the panel is small businesses that signed up for an AI-visibility tool, and EZY sells a tool that generates llms.txt files, which if anything gives them a reason to find the opposite.

The anecdote on the other side. In July 2025 Search Engine Roundtable reported that SEO Ray Martinez saw OpenAI hitting his llms.txt on a few sites, “pinging our servers every 15 minutes looking for freshness.” That’s one person’s logs from 2025, with no comment from OpenAI. The two larger 2026 datasets don’t show that pattern at scale.

Does having one get you cited? The citation study

SE Ranking (published October 15, 2026, by Yulia Deda) checked nearly 300,000 domains. 10.13% had an llms.txt. They compared that against how often each domain appeared as a source in sampled AI answers, using correlation tests and an XGBoost model. The finding: “There’s no correlation between AI citations and LLMs.txt.” Their model got more accurate on test data when they removed the llms.txt variable, which led them to say the file “might currently introduce more noise than value” as a predictor.

That’s correlation across a sample, from a company that sells AI-visibility tools, and the authors say the results are context-dependent. But it points the same way as the logs: the engines that produce AI answers barely open the file, and having it doesn’t track with being cited.

What each vendor says, from its own pages

Google. Its Search Central guide, Optimizing your website for generative AI features on Google Search (last updated July 10, 2026), is the clearest statement any vendor has made: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search.” And: “Doing so will neither harm nor help your site’s visibility or rankings in Google Search.” This covers AI Overviews and AI Mode. Before that, in April 2025, John Mueller, answering a Reddit thread, said that “AFAIK none of the AI services have said they’re using LLMs.TXT” and “To me, it’s comparable to the keywords meta tag” (per Search Engine Journal). In July 2025 Gary Illyes reportedly said at a Search Central Live event that Google doesn’t support llms.txt and isn’t planning to; I could only find that second-hand, through Search Engine Roundtable’s report of Kenichi Suzuki’s post.

The one Google product that checks for it. Chrome’s Lighthouse has an experimental Agentic Browsing category. Under Discoverability it lists: “llms.txt: Checks for the presence of a machine-readable summary at the domain root.” The page says the category is “experimental and based on proposed standards” and needs Chrome 150 or later. It’s an audit for sites that want browser agents to work well on them. It says nothing about Search or Gemini using the file.

OpenAI. Its crawler page (developers.openai.com, “Overview of OpenAI Crawlers”) describes OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot and how to control them with robots.txt. The only mention of llms.txt is a link to OpenAI’s own docs index: “For the complete documentation index, see llms.txt.” I found no OpenAI page saying its bots read other sites’ llms.txt.

Anthropic. Its help-center article on crawling lists ClaudeBot (training), Claude-User (fetches pages when a person asks Claude something) and Claude-SearchBot (search quality), and says they “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.” It doesn’t mention llms.txt. Anthropic does publish llms.txt files for its own docs: docs.claude.com/llms.txt and code.claude.com/docs/llms.txt both returned files on November 1, 2026, while anthropic.com/llms.txt returned a 404.

Perplexity. Its bots page describes PerplexityBot (search results, not training) and Perplexity-User (fetches for a user’s question). It mentions llms.txt only as an index of its own docs at docs.perplexity.ai/llms.txt. perplexity.ai/llms.txt wasn’t available when checked on November 1 (it returned an error).

So the pattern across all four: each publishes an llms.txt for its developer documentation, none says its crawlers or assistants read yours, and none puts one on its main marketing site. I’d read that as a fair summary of where the file is useful.

Where it does work: documentation read by coding agents

The spec itself (llmstxt.org, revised August 10, 2026 as v2) says the files “are used most heavily for software documentation”. The numbers behind that:

  • Coding agents are now a large share of docs traffic. Mintlify analysed 30 days of Cloudflare traffic across the docs sites it hosts (published April 3, 2026): about 790 million requests, 45.3% from identified AI agents. Claude Code alone made 199.4 million requests and Cursor 142.3 million. ChatGPT made 1.8 million. By July Mintlify put agents at 66% of measured traffic. These are Mintlify’s own figures, from a company that sells docs hosting.
  • llms.txt measurably helps those agents. In Mintlify’s benchmark (July 17, 2026), Claude Code and Codex answered 5 questions on each of 20 docs sites in four formats, 2,400 runs in all. Served raw HTML, the agents “kept reaching for .md and /llms.txt versions of pages” and hit dead links. Adding one link to llms.txt at the bottom of each markdown page cut dead URLs by about 90% (2.23 per task down to 0.11) and reduced token use. Accuracy stayed in the 90s in every format. As Mintlify puts it: “The win was efficiency, not correctness.” The benchmark is open-sourced on GitHub, so it can be rerun.
  • Big files need structure. Mintlify capped llms.txt at 100,000 characters, and large docs sites started dropping pages. Its rebuild split the file into a hierarchy of child files by navigation section; on 200 tasks across ten sites, agents finished 52.1% faster with 44.8% fewer tokens (Mintlify, September 1, 2026).
  • Developer tools read it directly. LangChain’s open-source mcpdoc (the repo is now archived) is an MCP server that hands a list of llms.txt files to Cursor, Windsurf, Claude Code or Claude Desktop, with a fetch_docs tool to read the pages they link. The spec lists more: plugins for VitePress and Docusaurus, Drupal support, a VS Code extension, and an MCP server.
  • The companies that build these agents ship llms.txt on their docs. Besides OpenAI, Anthropic and Perplexity above: ai.google.dev/gemini-api/docs/llms.txt, docs.stripe.com/llms.txt, vercel.com/llms.txt and developers.cloudflare.com/llms.txt all returned files on November 1.

One tension worth naming: Ahrefs saw AI bots never request a missing llms.txt, while Mintlify’s agents went looking for /llms.txt on docs sites that lacked one. Both can be true. Crawlers on general websites don’t probe for it; a coding agent working a task on a docs site sometimes does. That’s the split this whole article comes down to.

How to check your own logs

You don’t have to take any study’s word for it. Your server logs show who fetches the file. About 10 minutes.

1. Confirm the file is really served. Run:

curl -sI https://<your-domain>/llms.txt

You want 200 and a content-type of text/plain or text/markdown. If you get text/html, open the URL: many single-page-app hosts return the homepage for any path, which is a soft 404 (Ahrefs filtered these out of its study for that reason).

2. Count who requested it. On a server with Nginx or Apache in the standard combined log format:

zgrep -h '"GET /llms' /var/log/nginx/access.log* | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -20

This prints the user agents that requested /llms.txt or /llms-full.txt, most frequent first. Swap the path for /var/log/apache2/access.log* on Apache.

3. Compare against robots.txt. Run the same line with '"GET /robots.txt'. If GPTBot fetches robots.txt hundreds of times and llms.txt a handful, you’re seeing what EZY saw.

4. Look at the AI agents by name. To filter for the agents in this article:

zgrep -h '"GET /llms' /var/log/nginx/access.log* | grep -iE 'gptbot|oai-searchbot|chatgpt-user|claude|perplexity|meta-externalagent|googlebot'

5. Check they’re genuine. User agents can be faked. OpenAI publishes IP ranges for each bot as JSON files (openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json); compare the IPs in your matching lines. See verify-ai-bots for the other vendors.

On Cloudflare without server logs: go to AI Crawl Control, open the Metrics tab and filter by path /llms.txt. The filters cover date range, crawler, operator, hostname and path, and the “Most popular paths” table shows requests per path.

A note on reading the result: a fetch shows the file was downloaded. It doesn’t show that an AI answer used it, so treat the count as a ceiling, the way Ahrefs does.

What it costs to create and keep up

RouteWhat it takesWho it suits
Write it by handAbout 20 minutes with our template, then an edit whenever a key page is added, renamed or removedSmall sites with 10-30 pages worth listing
Yoast SEO (WordPress)Free; Yoast says it’s “Built into Yoast, free for everyone”, enabled in the site feature settings, and “generated and refreshed automatically”WordPress sites already on Yoast
AIOSEO, WixThe spec lists both as generating llms.txt automatically; I didn’t check their settings or plan limitsSites already on those platforms
Docs platforms (Mintlify, GitBook)Generated with the docs; Mintlify also adds markdown versions of every pageDeveloper docs
Static-site plugins (vitepress-plugin-llms, docusaurus-plugin-llms)A plugin in your build, so the file regenerates on every deployDocs built with VitePress or Docusaurus

The money cost is close to zero everywhere. The real cost is maintenance: a hand-written file goes stale the first time someone restructures the site and nobody remembers it exists. For a marketing site that’s a small risk, since almost nothing reads it. For docs it matters, because the agents that do read it will follow dead links. If you can’t regenerate it automatically, I wouldn’t hand-maintain it on a large docs site.

Don’t pay for “llms.txt optimization” as a service. Nothing in the evidence above suggests there’s anything to optimise for AI search.

How it fails

  • It goes stale. The file lists pages that have moved or no longer exist, and agents waste fetches on 404s, the exact problem Mintlify’s benchmark measured. Check: once a month, run every URL in the file through curl -s -o /dev/null -w "%{http_code} %{url_effective}\n" and look for anything other than 200.
  • It gets too big. A file listing thousands of pages floods an agent’s context, which is why Mintlify capped it and then split it into child files. If yours is over 100,000 characters, split it by section.
  • It’s served as a soft 404. Covered in step 1 above: the URL returns your homepage HTML with a 200, so tools think you have a file and agents get junk.
  • It’s treated as access control. llms.txt can’t block or allow anything. Listing a page in it doesn’t override a robots.txt Disallow, and leaving a page out doesn’t hide it. Access belongs in robots.txt and your firewall (see why robots.txt is advisory and llms-txt-complete-guide). Also note that the assistant fetchers may skip robots.txt anyway: OpenAI says of ChatGPT-User that “robots.txt rules may not apply”, and Perplexity says Perplexity-User “generally ignores robots.txt rules.”
  • It contradicts robots.txt. If llms.txt points to pages your robots.txt blocks for the same bot, compliant crawlers won’t fetch them, so the listing does nothing. Check: for each section you block in robots.txt, make sure llms.txt doesn’t link into it.
  • It says something your pages don’t. Mueller compared llms.txt to the keywords meta tag, and my read of why the search engines ignore it follows from that: a file where a site describes itself is easy to game, so they read the pages. Keep the descriptions in the file matching what’s on the pages.

Counter-evidence, both ways

Against my “it doesn’t help AI search” read:

  • Ray Martinez’s 2025 logs showed OpenAI fetching llms.txt every 15 minutes on a few sites.
  • In EZY’s data Meta-ExternalAgent fetched llms.txt more than robots.txt (193 against 172), and Googlebot fetched it 67 times. Meta could be doing something with it.
  • Google’s Lighthouse checks for it, and the spec’s August 2026 revision points to that audit as a sign of adoption. If browser agents start relying on it, the picture for ordinary sites could change.
  • Absence of a vendor statement isn’t proof of non-use. Fetch counts are a ceiling on use, but the vendors could also use files they fetch rarely.

Against my “it works for docs” read:

  • The strongest positive numbers come from Mintlify, which sells docs hosting with llms.txt built in. The benchmark is open-sourced, which helps, but I found no independent replication.
  • The benchmark measured efficiency. Accuracy was in the 90s with or without the file, so the agents got the right answer anyway, just more slowly and at higher token cost.
  • I found no study connecting llms.txt to a business result on a docs site, such as signups or fewer support tickets.

How to decide

  • Developer docs, an API, an SDK or a dev tool: yes. Generate llms.txt from your docs build, add markdown versions of pages, link to llms.txt from each page, split it if it passes 100,000 characters, and test it once with Claude Code or Cursor on a real question. This is where the measured benefit is.
  • SaaS product with public docs: put it on the docs subdomain, the way OpenAI, Anthropic and Perplexity do. Skip it on the marketing site unless it’s free.
  • WordPress or Wix marketing site, blog, local business: turn it on if your platform does it automatically, then forget it. Don’t write and maintain one by hand for AI search; Google says it won’t help, and the other engines barely fetch it.
  • Online store: same as a marketing site. I found no evidence that AI shopping answers read llms.txt.
  • Google rankings, on any site: it won’t help or hurt Google rankings, including AI Overviews and AI Mode; Google says it will “neither harm nor help” your Search visibility.
  • Anyone selling you llms.txt as a GEO service: no. What correlates with AI citations is covered in seo/ai-visibility.
  • Not sure: run the log check above for 30 days. If nothing but SEO tools fetches the file, you have your answer for your own site.

What I’d do this week

If you already publish an llms.txt, run steps 1-3 of the log check and look at who fetched it in the last month. If you run docs, also run the link check from “How it fails” and fix anything that isn’t a 200. If you’re a marketing site and haven’t added one, don’t make it a project: switch it on if your CMS offers it, and put the hour into something from seo/ai-visibility instead.

Sources

#llms-txt#geo#ai-crawlers#honest-assessment