If you have spent any time in SEO communities over the past year, you have seen the llms.txt pitch: add one small file to your website and AI systems like ChatGPT and Google's AI Overviews will finally understand your content. Some influencer posts go further and present it as the robots.txt of the AI era — an essential standard you are falling behind on.

We looked at the actual data. The gap between the hype and the evidence is one of the widest we have seen for any AEO tactic. This article documents what llms.txt is, what the adoption and crawler data actually show, what Google has said on the record, and the narrow case where the file genuinely does something — because there is one, and it is not the one being sold.

---

What is llms.txt?

llms.txt is a plain Markdown file placed at the root of a website, at yourdomain.com/llms.txt, containing a curated summary of the site's most important content with links. It was proposed in September 2024 by Jeremy Howard of Answer.AI to solve a technical problem: large language models have limited context windows, and raw HTML pages full of navigation menus, scripts, and cookie banners waste most of that window before the model reaches the actual content.

The proposal was never about search rankings. It was a technical fix for AI systems reading documentation. What happened next is a familiar pattern: documentation platforms adopted it, the SEO industry noticed, and a developer tool got repackaged as an AI search visibility tactic.

It is important to understand what llms.txt is not. It is not a standard. Unlike robots.txt, no governing body or major platform has committed to reading it. It is a proposal that anyone can implement and every AI company is free to ignore — and, as the data below shows, that is largely what they are doing.

---

Does Google use llms.txt for AI Overviews?

No, and this is not speculation — Google has said so on the record, repeatedly. Google's official guidance on optimising for generative AI features, updated in May 2026, explicitly states that llms.txt is not needed for AI Overviews, AI Mode, or any generative Search feature. A further clarification was added in June 2026 confirming the file is not required for Google Search.

Google's search team went further in public comments. John Mueller compared llms.txt to the keywords meta tag — a reference SEO veterans will recognise, since the keywords tag was abandoned by Google years ago because it was pure self-declaration with no verification. Gary Illyes confirmed Google is not pursuing support for the file.

There is a confusing wrinkle worth addressing because it fuels the hype. In May 2026, Chrome's Lighthouse tool added an experimental audit that checks whether a site provides llms.txt. Days after Google Search said the file was not needed. The contradiction is resolved by understanding that these are different teams solving different problems: the Lighthouse audit relates to agentic browsing — automated agents navigating websites to complete tasks — not to search rankings or AI Overview citations. Mueller himself described the file's role as a temporary aid for AI coding tools parsing developer documentation, not something most sites need.

---

What does the adoption and crawler data show?

Three independent data sources published in 2026 tell a consistent story.

Adoption is growing but still marginal. A year-long tracking study by Originality.ai monitoring over 3 million websites found llms.txt adoption grew from roughly 4,000 sites in June 2025 to just over 36,000 by May 2026 — an 8.8x increase. Separately, SE Ranking analysed 300,000 domains and found a 10.13 percent adoption rate. Growth is real. The base remains small.

Almost nothing reads the files. This is the finding that matters. The same Originality.ai study found that 97 percent of sites with a valid llms.txt file received zero requests for it — no bots, no humans, nothing. OtterlyAI ran a 90-day monitoring test on a domain with llms.txt implemented: out of more than 62,000 AI bot visits recorded, 84 requests targeted the file. That is roughly 0.1 percent of AI crawler traffic. A larger analysis by Limy across 500 million AI bot events found only 408 direct requests for llms.txt files. The file exists. The crawlers that drive citations — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — are not asking for it.

The most-cited sites mostly do not have one. In SE Ranking's dataset, among the 50 domains most frequently cited by AI systems, only one had an llms.txt file. Whatever is driving AI citations for the sites winning at AEO, it is not this file.

No provider has committed to it. OpenAI's crawler documentation directs site owners to robots.txt and does not mention llms.txt. Anthropic's crawler guidance takes the same position. Both companies publish llms.txt files for their own documentation — which sounds like endorsement until you notice that neither tells site owners their crawlers consume the file. Semrush ran a controlled study and found no statistical correlation between implementing llms.txt and improved performance in AI results.

---

Where does llms.txt actually work?

There is one environment where the file demonstrably gets used, and it is the environment it was originally designed for: AI developer tools and agents.

AI coding assistants — Cursor, Windsurf, Claude Code, GitHub Copilot — fetch llms.txt when pointed at documentation sites. The workflow is practical: the agent identifies which library owns a feature, fetches that library's llms.txt to get a clean map of the documentation, then pulls only the relevant pages before writing code. LangChain shipped an open-source MCP server built specifically around exposing llms.txt files to these tools. Anthropic recommends the file in its guidance on writing for agents, and OpenAI references it in its Agents SDK documentation.

This is the honest framing: llms.txt is agent-readiness infrastructure, not search visibility infrastructure. If your site is developer documentation, an API reference, or a product built to be consumed by automated agents, the file has a real, current, confirmed use case. If your site is a photography blog, a travel site, or a services business hoping for ChatGPT citations, the evidence says the file does nothing today.

---

Should you implement llms.txt anyway?

We did — and the reasoning matters more than the decision.

The cost is close to zero. Yoast SEO and Rank Math now generate the file automatically, which means for most WordPress sites implementation is a toggle, not a project. A file that costs five minutes and carries no penalty risk does not need a strong business case. It needs only a plausible future one.

The plausible future case is the sitemap precedent. Yoast founder Joost de Valk has made the most intellectually honest argument for implementation we have seen: XML sitemaps launched in June 2005 with exactly one search engine supporting them, and anyone analysing server logs in early 2006 would have correctly concluded almost nothing used sitemaps — and been completely wrong about the trajectory. Web standards get adopted because publishers ship first, which gives platforms a reason to look, which gives more publishers a reason to ship. Someone goes first.

That argument deserves to be taken seriously, and it also deserves its counterweight: most proposed standards die. For every sitemap there are a dozen keywords meta tags. The rational position is not "implement because it will matter" — it is "implement because the cost is trivial, and spend your actual optimisation hours on the things the citation data says work: direct-answer content structure, topical authority, entity consistency, and third-party presence on the platforms AI systems actually read."

We implemented llms.txt on two of our test sites in July 2026 using Yoast SEO, which generates the file automatically. We have set up Cloudflare WAF rules to monitor fetch events and will update this section with our own crawler data after a 30-day observation window.

---

How do you measure whether anything reads your llms.txt?

This is the part of the llms.txt conversation almost nobody does, and it is the only part that produces knowledge instead of hope. Do not trust general adoption studies — measure your own file.

The method is straightforward. Check your CDN or server access logs for requests to /llms.txt and /llms-full.txt, filtered by AI user agents: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended, Applebot-Extended. If your site runs on Cloudflare, the Analytics section breaks bot traffic down by user agent without touching raw logs. A more forensic option is embedding a honeypot link inside the file — a URL that appears nowhere else on your site — and monitoring for any hits to it. Any request to that URL proves something read the file and followed a link from it.

We are running exactly this measurement across the sites we manage and will publish the fetch data in a follow-up experiment article once we have a meaningful observation window.

Our honeypot monitoring is live as of July 2026. We will publish the fetch data here once the observation window closes. Check back or subscribe for the update.

---

What is the bottom line on llms.txt in 2026?

llms.txt is a real solution to a real problem — but the problem is AI agents parsing developer documentation, not AI search visibility. The evidence as of mid-2026 is unusually consistent: Google has said on the record it does not use the file, no major AI provider has committed to it, 97 percent of implemented files receive zero requests, and the sites winning AI citations mostly do not have one.

Implement it because it costs nothing and the sitemap precedent makes early adoption defensible. Do not implement it instead of the work that measurably moves citation probability: question-structured headings, direct answers in the first 60 words, FAQPage schema, content freshness, and genuine topical authority. The most expensive mistake in AEO right now is spending strategy hours on a file that nothing reads while competitors spend those hours on content structure that everything reads.

---