Guide · AI Crawlers

llms.txt: The Complete Guide to Telling AI Crawlers What to Index

By Vlad Radu, Founder of Wolfdesk · July 9, 2026

Most websites still rely on robots.txt to communicate with crawlers, but robots.txt was built for search engines — not LLMs. llms.txt is an emerging standard that tells AI crawlers which pages matter, what they contain, and how they fit together. Think of it as a curated table of contents for ChatGPT, Perplexity, Gemini, and Claude.

What Is llms.txt?

llms.txt is a plain-text or markdown file served at the root of your domain. It lists the URLs you want LLM crawlers to prioritize, grouped by topic and summarized in a few lines. Unlike a sitemap, it is human-readable and designed to help models decide which pages are worth indexing.

The standard was proposed by Jeremy Howard and the Answer.AI team in 2024 and is now supported by a growing set of AI crawlers. It sits alongside robots.txt and sitemap.xml; it does not replace them.

Why llms.txt Matters for AEO

Answer Engine Optimization depends on whether AI engines can extract and cite your content. If a crawler wastes its budget on obsolete pages or misses your most important guide, your citation rate drops. A well-written llms.txt:

Directs crawler attention to your canonical, citeable content.
Provides context that helps LLMs disambiguate similar pages.
Signals that your site is actively maintained for AI consumption.
Complements schema markup by adding a narrative layer to the URL list.

Which AI Crawlers Read llms.txt?

Support is evolving. As of mid-2026, the following crawlers are known to consume or respect llms.txt:

PerplexityBot — uses llms.txt to prioritize pages for its answer index.
Claude / Anthropic — reads llms.txt to scope which domains and pages are included in training data.
Open-source LLM crawlers — several community crawlers check for llms.txt before broad crawling.
OpenAI and Google-Extended — have announced support but still treat it as a secondary signal.

Even where support is partial, publishing llms.txt is low-risk and positions your site correctly as the standard matures.

How to Write Your llms.txt File

Create a markdown file at /llms.txt. Start with a short site description, then group links under headings. Keep summaries factual and concise — LLMs quote them.

# llms.txt — aeo-audit.app

> Free AEO audit tool that scores any URL on the technical, content, and authority signals that decide whether AI answer engines cite it.

## Product & guides

- [Home](https://aeo-audit.app/): Run a free AEO audit and see a citation-probability score, schema validation, and extractability report.
- [Blog](https://aeo-audit.app/blog): Field notes on Answer Engine Optimization and Generative Engine Optimization.
- [llms.txt guide](https://aeo-audit.app/blog/llms-txt-guide): How to write, deploy, and validate a llms.txt file for AI crawlers.

## AEO strategy

- [AEO strategy 2026](https://aeo-audit.app/blog/aeo-strategy-2026): The 2026 playbook for schema, llms.txt, E-E-A-T, and citation-rate measurement.
- [AEO vs GEO](https://aeo-audit.app/blog/aeo-vs-geo): Side-by-side comparison of Answer Engine Optimization and Generative Engine Optimization.
- [Check AI recommendations](https://aeo-audit.app/blog/how-to-check-if-chatgpt-recommends-your-website): How to check if ChatGPT, Perplexity, and Gemini are recommending your website.

## Legal

- [Privacy Policy](https://aeo-audit.app/privacy): GDPR data-processing and contact information.
- [Terms & Conditions](https://aeo-audit.app/terms): Service, price, and governing law.
- [Legal Notice](https://aeo-audit.app/legal): Operator details for Wolfdesk, Berlin.

What to Include in Your llms.txt

A useful llms.txt file is selective. It should not be a duplicate of your sitemap. Include the following:

A one-sentence site description that states what you do and who you serve.
Grouped links to your most important, canonical pages — 10–30 is usually enough.
A one-line summary for each link so the crawler knows why the page matters.
Exclusions or notes for sections that LLMs should ignore, such as old changelog pages or internal admin docs.

Common llms.txt Mistakes to Avoid

We see the same errors across audits on aeo-audit.app:

Dumping every URL from the sitemap. Curation is the point.
Writing marketing copy instead of plain descriptions. LLMs prefer factual summaries.
Forgetting to update the file when major pages change. Stale llms.txt entries hurt trust.
Serving it as HTML or PDF. Use plain text or markdown with a text/plain content type.
Blocking the same pages in robots.txt. If you want LLMs to read it, allow the crawlers.

How to Check If Your llms.txt Is Working

First, confirm the file is reachable at /llms.txt with a 200 status and a text content type. Then check that it is linked from your robots.txt or site footer if you want broad discovery.

For a deeper check, run an audit. aeo-audit.app detects whether your site has a llms.txt file, previews its contents, and flags missing or malformed entries as part of the overall AEO score. It also validates schema, extractability, and crawler access in one pass.

FAQ

Is llms.txt a replacement for robots.txt?

No. robots.txt tells crawlers where they can or cannot go; llms.txt tells LLM crawlers which pages are worth reading and how they relate to each other. Use both. robots.txt handles permissions; llms.txt handles curation and context.

Which crawlers currently read llms.txt?

PerplexityBot, Claude's crawler, and several open-source LLM crawlers already consume /llms.txt. OpenAI and Google have signaled support but do not rely on it as a primary signal yet. The file is a forward-looking AEO investment that pays off as adoption grows.

Where should I put my llms.txt file?

Serve it at the root of your domain as /llms.txt, over HTTPS, with a text/plain or text/markdown content type. Some crawlers also accept /llms.txt.md or /.well-known/llms.txt, but the root path is the canonical location and the safest default.

Can I update llms.txt without hurting rankings?

Yes. llms.txt does not affect traditional search rankings. It only influences how LLM crawlers select and summarize content. You can edit it freely to prioritize your most citeable, up-to-date pages.

Next step

Audit your site for llms.txt and AEO readiness

Run a free audit — llms.txt detection, schema validation, extractability score, and citation-probability rating.

Run free audit →