Deutsch Back to the city

GEO & AI search

llms.txt: What the file does and whether you need it

SERP TOWN Editorial · · 12 min read

Illustration: At the edge of a colorful cartoon city stands a red radio tower. In front of it, a small sign shows an open text list, with yellow arrows pointing from it to the houses of the city.

An llms.txt is a small text file on your website that tells AI programs in a few lines what the site is about and where the most important content is. The file sits in the root directory, for example at yourdomain.com/llms.txt, or in a subfolder, and is written in simple Markdown. The idea comes from a proposal by Jeremy Howard, and Google's testing tool Lighthouse currently calls the file optional. For its AI Overviews, Google says explicitly that you don't need any special AI text files. And OpenAI, Anthropic and Perplexity do not say on their crawler pages that they read the llms.txt of other websites. In our view, the file is therefore an optional extra that takes little work but promises nothing.

In this guide you will learn who proposed llms.txt, what goes in it, what the providers officially say about it, whether the file is worth it for your business and how to create it. Where we recommend something that is not stated this way in the original sources, we say so.

What is llms.txt and who proposed it?

llms.txt goes back to Jeremy Howard. He published the proposal "The /llms.txt file" on 3 September 2024. A second version followed in August 2026, which he explains on the "Changes" page. According to Howard, the proposal is open for community input, and it is discussed publicly on GitHub.

The idea behind it is simple. AI programs read websites all the time today, for example a chat assistant that opens pages to write an answer. But web pages are built for people. According to Howard, navigation, ads and scripts make it hard to get clean text out of them. On top of that, a whole website usually does not fit into what a language model can process at once. The llms.txt is therefore meant to offer a short, easy-to-read starting point: a few sentences about the website and links to the pages that really matter.

What matters is what the file is mainly used for, according to the proposal. Howard writes that llms.txt files are used most heavily for software documentation. There, coding assistants follow the links to find tutorials and technical references. But the same structure, he says, fits anywhere AI programs need a way into a website's content, explicitly including a business outlining its structure and policies. That "thousands" of sites have an llms.txt is Howard's own statement. We have not found an independent count.

What goes into an llms.txt and in which format

The file is always called llms.txt and is written in Markdown, a simple way of writing text with headings, lists and links. According to the proposal, it sits in the root directory of the website at /llms.txt or in a subfolder, for example /docs/llms.txt. A file then covers all pages under its path.

The proposal sets a fixed order:

  1. A title as the main heading with the name of the website or project. This is the only required part.
  2. A short summary as a blockquote, that is, a line that begins with a greater-than sign. It contains the key information needed to understand the rest of the file.
  3. Further explanations in normal paragraphs or lists, but without headings. This is where you can explain how the linked content should be understood.
  4. Link lists under subheadings. Each entry is a link with a name and an address, optionally followed by a colon and a short note about what is on the page.

According to the proposal, a section with the heading "Optional" is used by convention for links that an AI program can skip when it has little space.

For a business, this could look like the following, here as a made-up example with no real details: The title is the company name. Below it, a line such as "Carpentry workshop in Graz, builds made-to-measure furniture for private customers". Then a "Services" list with links to the pages about kitchens, wardrobes and how an order works, each with a short note. Then a "Contact" list with the link to the contact page. And at the very bottom, under "Optional", a link to the "About us" page.

The proposal goes one step further: pages with important information should also have a clean Markdown version at the same address with the ending .md. This is a second, separate proposal alongside llms.txt.

For the texts, Howard gives a few guidelines: concise, clear language, a brief description for each link and no ambiguous terms or unexplained jargon. He suggests testing the file by giving an AI program only the llms.txt and asking it questions about your own content.

llms.txt and robots.txt are not the same

You may already know robots.txt. This file sets which programs may visit your website. According to the proposal, the two files have different purposes: robots.txt tells automated programs what access is considered acceptable. The llms.txt is read on demand, when an AI program is looking up information for someone. Howard expected the file to be useful mainly at this moment of looking things up and less for training AI models.

According to the proposal, a sitemap, that is, the list of all pages for search engines, does not replace llms.txt either. The llms.txt is a short, curated overview.

In our view, this also means: an llms.txt does not block anything. If you want to allow or exclude AI crawlers, you still do that in robots.txt. One exception is fetches that a person triggers themselves, for example with a question in a chat: according to OpenAI and Perplexity, robots.txt rules may not apply to them. How this works is explained in our guide to GEO and AI search.

Who reads llms.txt? What the providers officially say

There are many claims about this online. We only looked at the providers' own pages, specifically where they explain how their programs visit websites.

Something else stands out: all three providers have an llms.txt for their own developer documentation. The OpenAI page points right at the top to its llms.txt as the complete documentation index, and so does the Perplexity page, explicitly "For AI agents". Anthropic (platform.claude.com/llms.txt) and Google for the Gemini API (ai.google.dev/gemini-api/docs/llms.txt) also provide such a file for their developer documentation. This shows that the providers use the format for their own pages. But it does not mean that ChatGPT, Claude or Perplexity evaluate your website's llms.txt when they write an answer. The official documents say nothing about this.

One Google tool does check the file, though: Lighthouse, a testing tool for websites whose help is on Google's "Chrome for Developers" site. In its help on audits for AI agents (as of May 2026), meaning AI programs that open websites and complete tasks on their own, it calls llms.txt an "emerging convention", that is, a practice that is just taking shape. According to Lighthouse, without the file AI agents may take longer to understand a website's structure. If the file is missing, Lighthouse marks the audit as "Not Applicable", because the file is optional at the moment. If the server returns an error when the file is fetched, the audit counts as failed. This is a statement about Lighthouse, not about Google Search.

What Google says about llms.txt

For Google Search, the situation is clear. On the page "AI features and your website" (as of December 2025), Google writes about AI Overviews and AI Mode: you don't need to create new machine readable files, AI text files, or markup to appear in these features. According to Google, there are no additional requirements for these features. The best practices for SEO remain relevant.

Google does not name llms.txt on this page. Google speaks generally of AI text files. We read this as also covering llms.txt. The statement applies only to the AI features of Google Search. What Google recommends instead are the basics: the page must be indexed and eligible to be shown with a snippet, important content should be on the page as text, and details such as those in your Google Business Profile should be up to date. How to set up and maintain your profile is explained in our guide to the Google Business Profile.

Does your business need an llms.txt?

There is no official answer to this. What is documented: according to Google, AI Overviews do not require AI text files, and OpenAI, Anthropic and Perplexity do not say on their crawler pages that they read other sites' llms.txt files. It is also documented that, according to its inventor, the file has so far been used mainly for software documentation.

Our recommendation (the sources do not say this): Do the basics first. Allow the crawlers of the AI search services in robots.txt, put the important things on your pages as text and keep your details up to date. In our view, this does more than any extra file. You can create an llms.txt afterwards if you have the time. The effort is small, and according to Lighthouse, AI agents may take longer to understand a website without the file. But an effect on visibility in AI answers is not documented anywhere.

In our view, the file is more worthwhile the more content you have that an AI program should be able to find directly:

By the way: serptown.com does not have an llms.txt at the moment either. When we checked our own site on 27 October 2026, serptown.com/llms.txt returned a 404 error, meaning "not found".

How to create an llms.txt

The steps follow the structure from the proposal.

1. Choose the most important pages

Write down which pages an AI program should know to describe your business correctly: services or products, how you work, contact, frequently asked questions. Our recommendation (the sources do not say this): Stick to a few really important pages. The proposal only says that the file should stay small enough to be read in one go.

2. Write the file in Markdown

Open a simple text editor and follow the order from above: name as the main heading, below it the short summary as a blockquote, then a few explanatory sentences if needed, then link lists under subheadings. For each link, briefly note what is on the page.

Our recommendation (the sources do not say this): Only put in the file what is also on your website. Name, location and services should read the same everywhere. And put nothing in it that should not be public.

3. Upload the file to the root directory

Save the file as llms.txt and upload it so that it can be reached directly under your domain, that is, at yourdomain.com/llms.txt. This is also how the Lighthouse help describes it.

Some website builders do this for you. Wix writes in its help on llms.txt that it automatically generates and maintains the file and that you can edit it or switch it off in the dashboard. Before that, according to Wix, you need to upgrade your site, connect a custom domain and enable search engine indexing. For WordPress, Yoast says on its llms.txt page that the plugin generates the file with one click. According to Yoast, this is included in the free version too, with no upgrade needed; you switch the feature on in the settings. This is not a recommendation for any particular tool. Check whether your system already offers something like this before you create a second file.

4. Check that the file can be reached

Open the address in your browser. You should see your text. According to its help, Lighthouse reports a failure if the server returns an error when fetching the file.

5. Test it with an AI program

The proposal itself suggests this test: give an AI program only your llms.txt and ask it questions about what you offer. If the answers are off, then in our view important links or explanations are missing from the file.

6. Keep it up to date

Our recommendation (the sources do not say this): Update the file when you add new important pages or delete old ones.

What llms.txt is not

llms.txt is not access protection, because allowing and blocking is handled by robots.txt. According to Google, its AI Overviews do not require AI text files, so in our reading they do not require an llms.txt either. And it is no promise that ChatGPT, Claude or Perplexity will mention your business. None of the three providers says on its crawler pages that it uses the file for this. Anyone who promises you more visibility in AI answers with an llms.txt is, in our view, promising more than the sources support. What is officially documented about AI answers is explained in our GEO guide in the SERP TOWN Library.

Your next step

Do you want to know whether your business has more important things to work on than an llms.txt? Take the 3-question check from SERP TOWN. It shows you which business type your company is and where your biggest levers on Google and in AI answers are. If you like, you can then send us a request: we look at your website and send you a short teaser with a few tips.

Your next step

Send us a short request. We look at your website for exactly these points and write back with what matters for your business.

Start the free SEO checkOpen in the SERP TOWN Library