AI Visibility · 7 min. read
What Is llms.txt? AI Crawlers, Governance and Machine-Legible Websites
llms.txt is an emerging way to guide AI systems towards a website’s most important resources, but its real value only appears within a wider machine-legibility strategy.
Donatas Tranauskis
Founder & Systems Architect · · Updated
Artificial intelligence is changing how organisations are discovered online. Instead of browsing long lists of search results, users increasingly receive direct answers through AI-assisted search engines and large language models (LLMs). This shift introduces a new challenge for website owners: ensuring that their expertise, services, and information can be accurately interpreted by machines.
In the traditional web, websites were designed primarily for human visitors and search engine crawlers. Today, a growing number of systems analyse websites to build internal knowledge models, generate summaries, and recommend businesses directly within AI responses. If a website is difficult for machines to interpret, its content may be misunderstood or overlooked entirely.
One idea gaining attention in developer communities is llms.txt. It was proposed by Jeremy Howard in September 2024 as a way for website owners to give language models a short, curated map of their most important content. It is a proposal, not a standard, but it highlights a broader trend: websites must increasingly consider how information is structured for both humans and machines.
1. What Is llms.txt?
An llms.txt file is a proposed file format designed to help large language models understand a website more efficiently. It is placed at the root of a domain, for example:
example.com/llms.txt
The file provides a curated list of the most important pages and resources on a website. Unlike a traditional sitemap that lists every available page, llms.txt focuses only on content that represents the core knowledge of a site.
The proposal uses Markdown, so the file is easy for both people and machines to read. Its structure is simple: a level-one heading with the site or project name (the only required part), a short summary in a blockquote, optional notes, and then sections headed with ## that list links in the form [name](url): notes. A section called "Optional" holds links a reader can skip when it needs a shorter version.
A typical llms.txt file may include:
- Site overview → A brief description of what the website represents.
- Key pages → Links to services, resources, or documentation.
- Educational content → Guides or articles that define the site’s expertise.
- Usage guidance → Information about how AI systems may use the content.
The intention is to remove unnecessary elements such as navigation menus, scripts, and visual layout components, allowing AI systems to focus on the information that matters most.
2. Why AI Systems Need Clear Signals
Modern AI systems do not simply index pages. They analyse websites to extract meaning, build structured knowledge, and generate answers based on multiple sources.
This process works best when websites provide clear signals about their structure and authority. Without those signals, AI models must rely on interpretation, which can sometimes lead to incomplete or inaccurate summaries.
Clear signals help AI systems understand:
- Which pages represent core expertise
- Which content is educational or authoritative
- Which pages are transactional or low informational value
- How different parts of a website relate to each other
Files such as sitemaps, structured data, and potentially llms.txt can help reduce ambiguity and improve how machines interpret a website.
3. The Current Reality
It is important to be clear about what llms.txt is today. The proposal page describes itself as a proposal open for community input, not a ratified standard.
On the search side, Google's guidance on AI features and your website (checked November 2026) says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." It does not mention llms.txt. We have not found documentation from a major search or AI provider stating that its systems use llms.txt when they crawl or answer questions about websites.
For website owners this means:
- Do not expect it to change rankings or AI answers. No provider we checked says it does.
- It does not replace structured data, a sitemap or technical SEO.
- It is cheap to publish and harmless if accurate, and it doubles as a plain-language index of your site for people and tools that choose to read it.
In other words, treat llms.txt as an optional extra, kept accurate, rather than an optimisation method.
4. Best Practices for Using llms.txt
For organisations that wish to experiment with this concept, a few principles can help ensure the file remains useful.
- Keep it concise → Focus on high-value content instead of listing every page.
- Use clear grouping → Organise links by sections such as services, documentation, and blog resources.
- Maintain accuracy → Update the file whenever key pages change.
- Prioritise authority → Include pages that demonstrate expertise and real work.
- Avoid sensitive pages → Exclude login areas, forms, and transactional pages.
5. What Content Should Go Into an llms.txt File?
The value of an llms.txt file depends entirely on the quality of the resources it highlights.
Content typically worth including:
- Homepage and site overview
- Core services or product pages
- Educational blog articles or guides
- Case studies demonstrating real work
- Trust pages such as About and company information
The aim is not to replicate a sitemap but to highlight the material that best defines the expertise and purpose of the website.
6. A Worked Example: Our Own llms.txt
We publish an llms.txt file on this site. You can read the whole file at www.dbeta.co.uk/llms.txt. Here is the opening, shortened:
# DBETA > DBETA (DBETA LIMITED, UK company 11917799) is a London web and platform > company, trading since 2019. It designs, builds and hosts websites and web > applications, and makes Bones, a web platform for building and running websites. This file lists the main pages of www.dbeta.co.uk with a one-line note on each. The HTML pages are the primary source; each also carries schema.org JSON-LD. ## Company - [About](https://www.dbeta.co.uk/about.html): Who we are, how we work and company details. - [Pricing](https://www.dbeta.co.uk/pricing-plans.html): Pricing plans and what each includes. ## Services - [AI Visibility Audit](https://www.dbeta.co.uk/ai-visibility-audit.html): Find out whether Google, Bing and AI assistants can tell who you are, what you offer and why to recommend you...
Line by line:
# DBETA– the one required part: the name of the site.- The
>blockquote – a one-paragraph summary of who we are. It uses the same facts as our About page and company records, so a reader gets an accurate summary without crawling the site. - The plain paragraph – tells the reader that the HTML pages are the primary source, and that this file and our JSON summaries add nothing those pages do not say.
## Company,## Services– grouped lists of links, each with a short note so the reader knows what the page is for before opening it.
The full file also points to our AIDI JSON files, compact machine-readable summaries of the same content. It is rebuilt each night from our AIDI files, so a new page appears in it the night after it goes live.
7. Why llms.txt Alone Is Not Enough
Although llms.txt is an interesting concept, it does not solve the broader challenge of machine-readable websites.
AI systems build knowledge models from multiple signals, including structured data, content quality, entity relationships, and technical architecture.
True machine-legibility usually requires:
- Structured schema data
- Clean internal linking
- Consistent entity definitions
- Machine-readable APIs or structured outputs
- Clear content hierarchy
In practice, llms.txt works best as one component within a broader architecture designed to make websites understandable to both humans and machines. We explain that wider picture in why AI systems prefer structured websites. To see how clearly your own site reads to search engines and AI assistants today, an AI visibility audit tests it directly.
8. Final Thoughts
The rise of AI-driven discovery is encouraging website owners to rethink how information is structured online.
Concepts such as llms.txt highlight the growing importance of clarity, structure, and authoritative content. While the format itself may evolve or even be replaced by future standards, the underlying principle will remain the same.
Websites that organise their knowledge clearly, demonstrate real expertise, and expose structured information are easier for machines to read correctly.
As AI search continues to develop, those foundations will likely matter far more than any individual optimisation technique.
In short
llms.txt is a proposed plain-text file that summarises a website for language models. It differs from robots.txt, which controls crawling, and sitemap.xml, which lists URLs, and it works as a supporting file alongside structured data, clear entity definitions and governed architecture rather than as a standalone AI tactic.
Questions about this article
llms.txt is an emerging text-based file used to highlight a website’s most important resources for large language models and AI systems. It is best understood as a curated guidance layer rather than a formal web standard or a guaranteed ranking signal.
No, not as a known ranking or indexing requirement. Today, Google still relies on established signals such as crawlability, structured data, content quality, internal linking and overall site architecture. llms.txt should be treated as experimental and supplementary.
No. robots.txt controls crawler access, sitemap.xml helps search engines discover URLs, and llms.txt attempts to guide AI systems towards the most useful resources on a site. They serve different purposes and should not be confused.
Not necessarily. Smaller websites may gain little from it on its own. It becomes more useful when a site has a larger knowledge footprint, multiple service areas, educational content, documentation, or a need for stronger machine-readable governance.
It should point to the content that best represents the site: core pages, high-value services, educational resources, case studies, trust pages and, where relevant, machine-readable endpoints. The goal is to reduce ambiguity for AI systems, not to duplicate every URL on the site.
Not directly. There is no evidence that llms.txt improves rankings on its own. Its value is strategic: it can support cleaner content governance and clearer machine interpretation when used alongside strong architecture, structured data and authoritative content.
Because AI systems do not rely on one file alone. They form understanding from multiple signals, including structured data, page content, entity consistency, internal linking, content quality and machine-readable outputs. llms.txt can support that process, but it cannot replace the underlying system.