What is llms.txt?

llms.txt is a proposed web file that gives large language models a short Markdown summary of a website and links to its most useful pages, served at /llms.txt. Jeremy Howard proposed it in September 2024. Google Search does not use it, and studies by SE Ranking and OtterlyAI found no measurable effect on AI visibility.

Why llms.txt matters less than you think

llms.txt shows up in many AI SEO checklists as the "robots.txt of AI". The data does not back that up.

SE Ranking analyzed about 300,000 domains. It found an llms.txt file on 10.13% of them and no correlation between having one and how often a domain is cited in LLM answers. Removing the llms.txt variable from its XGBoost model even made the model more accurate. OtterlyAI ran a 90-day log experiment instead: out of more than 62,100 AI bot visits to its test site, 84 requested /llms.txt, about 0.1%, while the average page on the site received around 265.

Google is explicit about its own engine. Its guide to generative AI features lists llms.txt among the things you can ignore, "as Google Search itself doesn't use them", and a June 2026 update to its changelog adds that these files "won't negatively or positively impact your visibility or rankings."

How llms.txt works

llms.txt works like a table of contents written for machines. The proposal, now in its second version on llmstxt.org, defines a plain Markdown file with a fixed order:

An H1 with the name of the site or project. It is the only required part.
A blockquote summary with the key facts a model needs to understand the rest.
Optional notes in paragraphs or lists, without headings.
H2 sections of links, each a list of [name](url) entries with an optional note. A section titled "Optional" holds links an agent can skip when it needs a shorter context.

The file sits at /llms.txt, or under a path such as /docs/llms.txt to cover only that part of the site. The proposal also asks sites to serve a clean Markdown copy of each useful page at the same URL with .md added.

llms.txt is built for inference, not training: an agent reads it on demand while helping a user, then follows the links it needs. Its author notes that coding agents reading software documentation are its heaviest users. The file grants or denies nothing. Access rules live in robots.txt.

Tooling is catching up even where search is not. Chrome's Lighthouse runs an llms.txt audit in its agentic browsing checks, but it only flags server errors and marks a missing file as not applicable, because providing one "is optional at the moment."

Example: an llms.txt file for a SaaS

Here is what a short file could look like for a fictional invoicing tool:

# Acme Invoice

> Acme Invoice is online invoicing software for freelancers and small agencies. Plans start at $9 per month.

Acme Invoice does not offer payroll or accounting.

## Product

- [Pricing](https://acme-invoice.example/pricing.md): Plans, limits and annual discounts
- [Features](https://acme-invoice.example/features.md): Invoices, reminders, multi-currency

## Help

- [Export invoices to CSV](https://acme-invoice.example/help/export-csv.md): Step-by-step guide

## Optional

- [Changelog](https://acme-invoice.example/changelog.md)

Now take a question a freelancer might ask an AI assistant with web access: "How do I export my Acme Invoice invoices to CSV?" An agent that supports llms.txt can fetch the file, spot the export guide under Help and load its Markdown version instead of parsing the full help center. An engine that does not use the file, which includes Google Search, crawls and ranks the regular pages as usual. Either way, the file is only as good as the pages behind its links.

llms.txt vs robots.txt

The two files share a location and an extension, and some guides describe llms.txt as a way to allow or block AI crawlers. It is not.

	llms.txt	robots.txt
Purpose	Point AI agents to your most useful content	Tell crawlers which URLs they may fetch
Status	Community proposal	IETF standard (RFC 9309)
Main readers	Some AI agents, mostly for documentation	Search and AI crawlers that honor it
Can opt you out of AI training or search	No	Yes, crawler by crawler
Measured effect on AI citations	None found	Blocking OAI-SearchBot keeps a site out of ChatGPT search answers

How to decide whether to publish llms.txt

An llms.txt file is cheap to publish and harmless, so the real question is its priority against the rest of your technical GEO work. Work in this order:

Check access first. Make sure robots.txt, your CDN and your firewall let AI crawlers through, such as OAI-SearchBot for ChatGPT search and Googlebot for Google AI Mode. As SE Ranking points out, an llms.txt file won't override restrictions in robots.txt.
Publish it where agents use it. Documentation, APIs and developer products benefit most.
Keep it short and current. A stale price or a dead link misleads the agents that do read the file.
Measure answers, not the file. Server logs show whether bots fetch /llms.txt. Only the answers show whether you are cited.

Hawkscan's action plan points you to fixes that change what engines can use, such as an AI crawler your site blocks or a page that cites your competitors but not you.

Common questions

Does ChatGPT use llms.txt?

OpenAI has not said that ChatGPT uses llms.txt to choose or cite sources. Its crawler documentation tells site owners to manage ChatGPT search access through robots.txt and OAI-SearchBot. OpenAI does publish an llms.txt file for its own developer documentation, as Anthropic and Google's Gemini API docs do, which is a different use: helping coding agents find pages.

Can llms.txt hurt my rankings?

No. Google says llms.txt files won't affect visibility or rankings in Google Search either way, and that keeping one for other services that use it is fine. The only real risk is indirect: an outdated file can send the agents that read it to old prices or pages that no longer exist.

Does llms.txt stop AI models from training on my content?

No, llms.txt contains no access rules. To opt out of training, disallow training crawlers in robots.txt, such as GPTBot for OpenAI or ClaudeBot for Anthropic. Their search crawlers are separate, so you can stay eligible for ChatGPT search or Claude's search results while opting out of training.