Where robots.txt says 'do not go there', llms.txt says 'start here': a markdown index of the site's most quotable content — docs, definitions, key pages — in a format language models read natively. Proposed in 2024, adopted by documentation-heavy sites first, unproven in crawler behavior but nearly free to ship.
The honest framing: no major AI crawler has committed to reading it, so it is a bet, not a standard. The reasoning behind placing it anyway — the file is a curated summary of your best content in the format machines prefer — means the same effort also produces the markdown mirrors and structured data that definitely get read. Cheap experiment, real byproducts.
Related terms
Answer engine optimization (AEO)
Structuring content so that answer engines — chat assistants and AI overviews — can find it, quote it and attribute it correctly.
robots.txt
A plain-text file at the site root telling crawlers what they may fetch — an instruction to machines, not a lock on a door.
Structured data
Machine-readable markup — usually JSON-LD written in the schema.org vocabulary — that tells search and answer engines what the entities on a page are.
The bench this belongs to
Web performanceA rebuild that loads on a phone on mobile data, in the country your customers are actually in. Core Web Vitals are the measurement, revenue is the point.
