Skip to content
Search

Llms.txt

lastmod 2025-07-14created 2025-04-112 edits

A proposed standard for providing website usage information to LLMs

About

  • Websites are used not only to provide information to people but also to LLMs
  • LLMs are used to enhance the development environments coders use, including the option of gathering information about programming libraries and APIs from website documentation
  • Providing information for LLMs differs slightly from providing it for people, but overlaps in many ways
    • They generally prefer information in a more concise format
    • They can take in a lot of information quickly, so gathering all key information in one place helps
    • LLM context windows are too small to handle an entire website, and converting complex HTML pages into LLM-friendly plain text documents is difficult
  • Therefore, a way is needed to identify the most important information to give AI assistants and provide it in the most suitable format

Proposal

  • Proposes that those who want to provide LLM-friendly content add a /llms.txt file to their site
    • This is a Markdown file that provides brief background information and guidance, with links to Markdown files that provide more detailed information
    • It can be used to provide the information needed to use a library, or as part of research to learn about a person or organization
  • llms.txt Markdown can be read by both people and LLMs, while also allowing processing by precise formats
  • llms.txt files can be used in various scenarios
    • For software libraries, they provide a structured overview of the documentation so LLMs can easily find specific features or usage examples
    • Corporate websites can outline their organizational structure and key information sources
    • Information about new legislation and its necessary background and context can be curated in an llms.txt file to help stakeholders understand it
    • Personal portfolio or CV websites can help answer questions about the individual
    • E-commerce sites can briefly describe product categories and policies
    • Educational institutions can use them to summarize course offerings and resources

llms.txt file format

  • The llms.txt file format provides information structure using Markdown, which language models understand easily
  • The file uses Markdown instead of a traditional structured format (e.g. XML) because its main readers are language models and agents
  • llms.txt files can be read with standard programming tools and are organized according to a specific format
    • H1 header: contains the name of the project or site, and is the only required section.
    • Blockquote: contains a short summary of the project, providing the key information needed to understand the rest of the file.
    • Markdown sections: provide detailed information and may contain several types of sections such as paragraphs and lists, but no additional headings.
    • File list sections delimited by H2 headers: contain lists of URLs that provide additional information; each entry is written as a Markdown link name and can optionally include a description.

Example: 

# 프로젝트 제목

> 프로젝트에 대한 선택적 설명이 여기에 들어감

선택적 세부 정보가 여기에 들어감

## 섹션 이름

- [링크 제목](https://링크_URL): 선택적 링크 설명

## Optional

- [링크 제목](https://링크_URL)
  • "Optional" section: has a special meaning; its URLs can be skipped if a shorter context is needed. Contains secondary information that can often be skipped.

Coexisting with existing standards

  • llms.txt is designed to work alongside existing web standards
  • Whereas sitemaps list all pages for search engines, llms.txt provides a curated overview for LLMs (large language models)
  • llms.txt complements robots.txt and can provide context for allowed content
  • It can also reference the structured data markup used on the site, helping LLMs understand and interpret it
  • Standardizes the file path in the same way as /robots.txt and /sitemap.xml
    • robots.txt and llms.txt serve different purposes
      • robots.txt: used by automated tools (e.g. search indexing bots) to understand the scope of allowed site access
      • llms.txt: mainly used when a user explicitly requests information on a specific topic. Useful, for example, when including a coding library's documentation in a project or asking a chatbot with search capabilities for information
    • llms.txt is expected to be useful mainly for inference. It is expected to be used at the moment a user needs help, not for training. However, if llms.txt becomes widely used, this information could possibly be used in future training as well
  • Differences from sitemap.xml
    • sitemap.xml: lists all indexable human-readable information on a site
    • llms.txt is not a replacement for sitemap.xml:
      • It may not include LLM-readable versions of pages.
      • It does not include URLs of external sites that may be useful for understanding the information.
      • It generally includes documents too large to fit in an LLM's context window, along with much information unnecessary for understanding the site.
    • llms.txt can work as a tool optimized for language models to access specific information and understand context

See also

Favorite site

Online Tools

author Zer0 sitemap