The llms.txt protocol faces a fundamental adoption problem eight months after its September 3, 2024 launch. According to analysis from Ahrefs, “no major LLM provider currently supports llms.txt. Not OpenAI. Not Anthropic. Not Google.” This lack of industry backing undermines the protocol’s core premise of helping large language models access structured website content.
Jeremy Howard’s proposal gained attention for addressing technical limitations in AI content processing. “Large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety,” Howard stated in his specification. However, the solution requires cooperation from AI companies that have shown little interest in implementing the standard.
Get the PPC Land newsletter ✉️ for more like this.
Subscribe
Summary
Who: Jeremy Howard proposed llms.txt in September 2024, while major AI companies including OpenAI, Google, Anthropic, and Meta have ignored the standard. Marketing professionals and SEO tool vendors continue debating implementation despite platform non-adoption.
What: llms.txt is a proposed protocol using markdown files to help AI systems access structured website content. However, no major language model providers parse these files, making implementation purely speculative without functional benefits.
When: The protocol launched September 3, 2024, with analysis through 2025 consistently showing zero adoption by AI platforms. Server log evidence confirms AI crawlers do not request llms.txt files during website visits.
Where: While llms.txt files can be hosted at website root directories globally, the lack of consumption by target AI systems makes location irrelevant. Community directories list hundreds of implementations that serve no functional purpose.
Why: The protocol aimed to address AI content processing limitations and give website owners control over AI interactions. However, without platform adoption, llms.txt represents wasted effort while marketing professionals face real traffic challenges from existing AI features that ignore the standard.
Cloudflare launches pay per crawl to monetize AI content access
Cloudflare introduces pay per crawl service allowing content creators to charge AI crawlers for access.
The disconnect between promotion and practical utility has created confusion in the marketing community. SEO tool provider Semrush began flagging missing llms.txt files as site issues, despite the protocol’s lack of functional impact. According to discussions on professional forums, this approach frustrates marketing professionals who question why they should “incentivize people to get everything they need from an AI response and NOT visit their website.”
Technical analysis reveals the scope of non-adoption. According to Google’s John Mueller, “none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it).” This server log evidence contradicts claims that AI systems parse llms.txt files during content crawling processes.
Mueller compared the protocol to obsolete web standards. “To me, it’s comparable to the keywords meta tag – this is what a site-owner claims their site is about,” he explained. The comparison highlights how website owners can create declarations about their content that search systems may completely ignore.
Industry experts have documented the protocol’s speculative nature. Ryan Law, Director of Content Marketing at Ahrefs, characterized the current state bluntly: “llms.txt is a proposed standard. I could also propose a standard (let’s call it please-send-me-traffic-robot-overlords.txt), but unless the major LLM providers agree to use it, it’s pretty meaningless.”
The implementation gap becomes more significant considering AI systems’ growing impact on web traffic. Recent studies show Google’s AI Overviews reduced organic clicks by 34.5%, while 80% of companies actively block AI crawlersaccording to HUMAN Security’s analysis. Website owners face traffic challenges without effective protocols for managing AI interactions.
The protocol’s technical design follows established web standards patterns. According to Howard’s specification, llms.txt uses markdown formatting with H2 headers organizing links to key resources. The file structure includes project summaries and categorized content listings designed for AI interpretation. However, sophisticated technical design proves irrelevant without platform adoption.
Several technology companies have published llms.txt files, including Anthropic, Cloudflare, and developer platform Mintlify. Yet these implementations represent performative adoption rather than functional integration. According to the documentation, “Anthropic: Publishes its own llms.txt, but doesn’t state that its crawlers use the standard.” Even companies creating their own llms.txt files do not commit to parsing others’ implementations.
The adoption challenge extends beyond major platforms to specialized AI services. According to the analysis, “Meta (LLaMA): No public crawler or guidance, and no indication of llms.txt usage.” Perplexity, ChatGPT, and other popular AI systems similarly provide no evidence of llms.txt support in their crawling behavior.
Marketing tool vendors continue promoting llms.txt despite functionality gaps. According to Search Engine Land analysis, “Semrush is flagging a lack of llms.txt file as an issue” while “tools like that flag a lot of things that aren’t really issues.” This creates additional work for marketing teams without corresponding benefits.
Professional SEO practitioners express skepticism about implementation value. According to forum discussions, “everyone’s scrambling in a dark room where nothing’s clearly visible” regarding AI optimization strategies. Brett Tabke, CEO of Pubcon and WebmasterWorld, argued that existing protocols already serve llms.txt’s intended purpose: “XML sitemaps and robots.txt already serve this purpose.”
The protocol’s community-maintained directory at directory.llmstxt.cloud lists implementations across various industries. However, these listings represent hope rather than functionality. Website owners create llms.txt files anticipating future adoption that may never materialize from AI platform providers.
Technical implementation remains straightforward despite uncertain utility. According to the specification, llms.txt files use simple markdown syntax with standardized header structures. Multiple generation tools have emerged, including WordPress plugins and automated crawlers. The ease of creation contrasts sharply with the lack of consumption by target systems.
Research methodology from Ahrefs revealed the protocol’s practical limitations. “AFAIK none of the AI services have said they’re using LLMs.TXT,” according to their analysis. Server log examination confirms that AI crawlers do not request llms.txt files during website visits, indicating zero actual utilization.
The economic context makes adoption uncertainty more problematic for website owners. Analysis shows AI search visitors convert at 4.4 times higher rates than traditional organic visitors. Without working protocols for AI optimization, marketing professionals lack effective strategies for capturing this higher-value traffic.
Industry observers note the irony in promotional approaches. Rob Garner’s analysis for Search Engine Land highlighted that “there is no evidence that llms.txt improves AI retrieval, boosts traffic, or enhances model accuracy” while marketing tools simultaneously recommend implementation. This disconnect between evidence and promotion characterizes much AI optimization guidance.
The broader AI governance landscape complicates protocol adoption prospects. Google’s AI Overviews face significant spam problems, while Microsoft emphasizes content audits over protocol implementation for AI search optimization. Major platforms focus on algorithmic solutions rather than standardized protocols.
Marketing professionals face resource allocation decisions amid uncertain returns. According to professional discussions, implementing llms.txt requires time and maintenance without demonstrated benefits. “Right now, most LLM vendors treat llms.txt as an interesting idea, and not something that they’ve agreed to prioritize and follow,” according to technical analysis.
The specification’s author acknowledged implementation challenges. Howard’s documentation emphasizes that “this proposal does not include any particular recommendation for how to process the llms.txt file, since it will depend on the application.” This flexibility creates compatibility uncertainty even if platforms eventually adopt the standard.
Alternative approaches gain attention as llms.txt adoption stalls. SEO expert Aleyda Solis released comprehensive AI search optimization guidelines focusing on content structure and crawlability improvements. These strategies address AI optimization without requiring new protocol adoption.
The protocol’s future depends entirely on major platform decisions. According to industry analysis, “creating an llms.txt is not the same as enforcing it in crawler behavior.” Website owners who implement llms.txt files essentially create documentation that AI systems ignore during content processing.
Current evidence suggests llms.txt will remain a theoretical standard unless AI companies commit to implementation. Marketing professionals must consider whether speculative protocol adoption justifies resource investment while facing demonstrable traffic challenges from existing AI features.
Timeline
• September 3, 2024: Jeremy Howard publishes llms.txt specification with no platform adoption commitments
October 2024: Major AI platforms show no indication of llms.txt implementation despite protocol availability
March 28, 2025: Search Engine Land documents widespread non-adoption among LLM providers
April 22, 2025: Ahrefs confirms no major AI services check for llms.txt files in server logs
May 14, 2025: Google focuses on spam prevention rather than protocol adoption for AI features
May 25, 2025: Microsoft recommends content audits over protocol implementation for AI optimization
June 16, 2025: Industry experts promote alternative AI optimization strategies bypassing llms.txt
July 2025: 80% of companies block AI crawlers entirely, making protocol adoption irrelevant