A proposed standard would guide large language models such as ChatGPT, Gemini, and Claude to relevant web content, helping online shops show up in AI-powered shopping.
The llms.txt file is akin to both the familiar robots.txt and an XML sitemap. For ecommerce merchants, the new, two-part standard could help share product information and other content directly with AI and the consumers who increasingly shop on those platforms.
Consumers increasingly shop on generative AI platforms. ChatGPT recommended these shoes and stores when prompted, “What are the best trail-running shoes?”
AI Visibility
Table of Contents
Jeremy Howard, the co-founder of Fast.ai and Answer.AI, proposed the llms.txt standard in September 2024. The aim is to help LLMs process and understand content from large, complex websites.
Thus, if adopted, the (nonbinding) standard could be a way for websites, like ecommerce businesses, to influence AI visibility.
The llms.txt proposal is gaining traction thanks to some support from Perplexity and Anthropic (Claude), and growing interest from the web community.
What Is llms.txt?
Llms.txt is a machine-readable file for AI. It signals to AI systems which content (i) is available for their use and (ii) is relatively easy to summarize and cite.
The llms.txt concept is similar to a robots.txt file or a sitemap. It can “allow” or “disallow” various AI tools from using portions of a site’s content.
User-Agent: openai Disallow: / User-Agent: anthropic Disallow: /private/ User-Agent: * Allow: /
The LLMs scraping a site do not have to follow these directives, but the allow and disallow statements are a way of signaling which content could be useful.
For example, an ecommerce site might “disallow” content in a shopping cart or checkout flow, encouraging an LLM’s bot not to waste its time.
User-Agent: * Disallow: /cart/ Disallow: /checkout/
Similarly, the file could push the AI toward content it can use, allowing paths to shipping guides, blog posts, and product descriptions.
User-Agent: openai Allow: /shopping-guides/ Allow: /blog/ Allow: /products/
This allow-disallow feature is the most cited among observers, but the file’s primary purpose is to provide content.
Structure
Many examples in the llms.txt directory follow the following pattern in Markdown, a popular, lightweight text markup language:
- H1 title
- Blockquote summary
- Sections (## Docs / ## Examples / ## Optional)
- Bullet links with [title](URL): description for .md content
Here is an example of an online store that sells science fiction movie memorabilia.
# sci-fi-memorabilia-example.com > Premium science fiction collectibles, props, replicas, and guides. ## Docs - [Star Wars Lightsaber Replica](.../star-wars-lightsaber-replica.md): Features, display use, product link. - [Top 10 Sci-Fi Collectibles](...).md - [Blade Runner Blaster Guide](...).md ## Optional - [Blog on cosplay tips](...).md User-Agent: openai Allow: /blog/ ... Contact: support@sci-fi-memorabilia-example.com Policy: https://sci-fi-memorabilia-example.com/policy
This file tells the LLM something about the website and offers alternative content in the form of linked Markdown (.md) files.
This format is similar to an XML sitemap that leads the crawler to various content pages.
Simplified Content
An llms.txt file for an ecommerce site could feature many links to Markdown versions of product detail pages.
In the example above, note the link to a page for a “Star Wars Lightsaber Replica.”
## Docs - [Star Wars Lightsaber Replica](.../star-wars-lightsaber-replica.md): Features, display use, product link.
The associated .md file contains no navigation or JavaScript content, but it does include the information an LLM would need to summarize, share, and cite the product. The machine-readable file is clean, concise, and factual.
Here is an example of what the AI would find in one of these product.md documents.
# Star Wars Lightsaber Replica – Luke Skywalker Edition > A premium collectible replica of Luke Skywalker's green lightsaber from *Return of the Jedi*. Built for display, cosplay, or collection. ## Overview This officially licensed replica features an all-metal hilt, removable LED blade, and motion-sensitive sound effects. It closely follows the design used on screen and comes with a stand and collector packaging. Ideal for fans, collectors, and cosplayers seeking accuracy and quality. ## Key Features - Green LED blade with ignition and clash effects - All-metal hilt modeled after Luke Skywalker's weapon in *Return of the Jedi* - Removable blade for wall-mounted display - Display stand and collector box included - Powered by AA batteries (not included) - Officially licensed by Lucasfilm ## Who It's For - Star Wars collectors seeking screen-accurate replicas - Cosplayers attending conventions or fan events - Gift buyers shopping for Star Wars fans - Sci-fi enthusiasts building a prop display ## Source https://sci-fi-memorabilia-example.com/products/star-wars-lightsaber-replica ## Tags star-wars, lightsaber, luke-skywalker, collectible, replica, prop ## Last Updated 2025-07-12
Extracting the document’s structure, we find:
- Clear title,
- Short overview paragraph (what it is, why it matters),
- Structured sections (features, uses, tags),
- A canonical source link (so the LLM can cite the store’s product page).
During Interface
A key feature of llms.txt is aiding LLMs at the “time of interface,” when a user is interacting with the chatbot.
A user asks a question about replica lightsabers. The AI visits the sci-fi memorabilia site, consumes the Markdown file, and quickly responds.
Think of the product Markdown file as CliffsNotes for the LLM!
AI Discoverability
Llms.txt resembles early optimization efforts in organic search. Similarly, the standard could help optimize for chatbots and answer engines — call it “LLM SEO.”
Llms.txt may or may not get traction, but it’s evidence that both LLM companies and website operators seek ways to aid AI in indexing and referencing pages.
It could shape the future of AI discoverability.
