Every serious bot reads /robots.txt first: which paths are disallowed, where the sitemap lives, sometimes which bots specifically are welcome. 'Disallow: /admin' says 'please do not crawl this' — it keeps crawlers out of areas that waste their budget, like filtered search pages and internal tools.
Two things it is not: it is not security — the file is public and names exactly what you hid, so sensitive paths still need auth — and it is not a guarantee: Google honors it, malicious bots ignore it, and a 'Disallow' does not remove a page from the index if it is already there. For that you want noindex on the page itself.
Related terms
XML sitemap
A machine-readable list of every URL worth indexing — last-modified dates included — so crawlers find all of it, fast.
Indexing
A search engine adding a page to its searchable database — crawling fetches it, indexing makes it findable; unindexed is invisible.
SEO
Search engine optimization — making pages findable, indexable and worth ranking: technical health plus content people actually want.
The bench this belongs to
Web performanceA rebuild that loads on a phone on mobile data, in the country your customers are actually in. Core Web Vitals are the measurement, revenue is the point.
