27 October 2026

What is llms.txt, and does it help you get recommended by AI?

llms.txt is a plain text file, written in Markdown, placed at the root of a website (yoursite.com/llms.txt) to give AI tools a short, readable summary of the site and links to its most useful pages. It was proposed in 2024 as a simple standard. It is cheap to add and can help tools that read it, but no major AI assistant has said it uses llms.txt to decide which brands to recommend, so treat it as a small extra, not a strategy.

Where llms.txt came from

The format was proposed in September 2024 by Jeremy Howard of Answer.AI. The idea is simple: web pages are full of navigation, scripts and layout that language models do not need, and a context window is limited. A short Markdown file that says what the site is and points to the key pages is easier for a model to use.

It sits alongside robots.txt and sitemap.xml but does a different job. robots.txt says what crawlers may fetch. sitemap.xml lists every page for search engines. llms.txt is a curated summary for a reader that has little time.

What goes in the file

The proposal keeps it light:

A title (the name of the site or project) as a heading.
A one-paragraph summary of what the site is, written plainly.
Sections of links to the pages that matter most, each with a short note on what it contains.
Optionally, a fuller companion file (often called llms-full.txt) with the text of the key pages in one place.

Who actually reads it

Some developer tools and documentation assistants read llms.txt files, and many software documentation sites publish one so coding assistants can find the right pages. That is where it is most useful today.

For the big consumer assistants, the picture is less clear. None of the major providers has said publicly that it uses llms.txt when choosing which products to name in an answer. Google, for example, has said its AI search features rely on its normal crawling and indexing. So an llms.txt file is unlikely to move a brand into a ChatGPT or Google AI answer on its own.

Should you add one anyway?

For most companies it is worth the half hour. It costs almost nothing, it forces you to describe your product in one clear paragraph (which is useful everywhere), and it helps the tools that do read it. Just do not expect it to change what assistants say about you.

What does affect AI answers is the same material that AI tools read across the web: a clear description of what you do on your own site, accurate listings on the comparison pages that rank for your category, and specific pages for the use cases buyers ask about. We publish our own llms.txt at shortlist.bobwork.ai/llms.txt, listing every leaderboard and guide.

How to know if anything changed

Adding a file, or any other change, is only worth it if you can see the effect. Pick the buyer questions for your category, ask each one several times on each assistant before the change, and again a few weeks after. Compare the share of answers that name you, with an error bar, so normal day-to-day variation is not mistaken for progress.

Questions

Is llms.txt an official standard?

It is a community proposal, not a standard from a standards body or from the AI providers. Adoption is voluntary and uneven.

Does llms.txt replace robots.txt or a sitemap?

No. robots.txt controls crawling and the sitemap lists pages for search engines. llms.txt is an extra summary for AI tools. Keep all three.

Can llms.txt block AI tools from using my content?

No. It is a summary, not a permission file. Crawler access is controlled in robots.txt.