The file lists user agent groups and allow or disallow rules, and it can point to your sitemap. Well behaved crawlers, including Googlebot and the major AI crawlers, check it before fetching. It is a set of requests, not enforcement, so it is not a security measure and anything genuinely private needs authentication. The important subtlety is that disallowing a URL stops crawling but does not guarantee it stays out of the index, since a page blocked from crawling can still be indexed from external links. To keep a page out of results, use a noindex tag and leave it crawlable.
Most service sites need almost nothing in this file: block admin and cart paths, point to the sitemap, and stop there. The reason to know it exists is that a single stray line can be catastrophic. Disallow slash blocks your entire site, and that line survives from development environments into live launches more often than anyone would like. Check yours at yoursite.com/robots.txt after any rebuild or host migration, and also check that you have not blocked the AI crawlers you would rather be visible to.
Send the site over. We will tell you what is broken, what you could rank for, and whether AI search knows you exist. No charge and no pitch deck.
