Where robots.txt goes
The file is normally available at https://example.com/robots.txt. Rules apply to the host, protocol, and port where the file is served, so a robots.txt file on one host does not automatically control every subdomain.
Core directives
User-agent identifies the crawler group. Disallow describes paths the group should not crawl. Allow can permit a more specific path inside a broader blocked path. Sitemap points crawlers to an XML sitemap URL.
User-agent: *
Disallow: /private/
Allow: /private/help/
Sitemap: https://example.com/sitemap.xmlrobots.txt vs. noindex
A common mistake is treating Disallow as a guaranteed “do not index” directive. robots.txt primarily controls crawling. If the goal is to request that a crawlable page not appear in search results, an indexing directive such as noindex is the relevant control.
robots.txt is not security
Do not use robots.txt to protect private documents, admin areas, customer data, or staging sites. The file is public, and compliant crawler behavior is not the same thing as authentication. Use real access controls for sensitive content.
Rule matching
Crawlers select the most specific applicable User-agent group. When the same product token appears in multiple groups, their rules are combined. Matching then uses the most specific Allow or Disallow path; when equally specific Allow and Disallow rules conflict, Allow wins.