# robots.txt

> A plain-text file at the site root telling crawlers what they may fetch — an instruction to machines, not a lock on a door.

Every serious bot reads /robots.txt first: which paths are disallowed, where the sitemap lives, sometimes which bots specifically are welcome. 'Disallow: /admin' says 'please do not crawl this' — it keeps crawlers out of areas that waste their budget, like filtered search pages and internal tools.

Two things it is not: it is not security — the file is public and names exactly what you hid, so sensitive paths still need auth — and it is not a guarantee: Google honors it, malicious bots ignore it, and a 'Disallow' does not remove a page from the index if it is already there. For that you want noindex on the page itself.

## Related terms

- https://dfieldsolutions.com/en/glossary/xml-sitemap.md
- https://dfieldsolutions.com/en/glossary/indexing.md
- https://dfieldsolutions.com/en/glossary/seo.md

---

Source: https://dfieldsolutions.com/en/glossary/robots-txt
DField Solutions — Dunakeszi, Hungary — dezso@dfieldsolutions.com
Booking: see https://dfieldsolutions.com/en/contact