Robots.txt Guides

Robots.txt for Subdomains: How We Check Scope Before Changing Rules

Our workflow for auditing robots.txt scope across root domains, subdomains, protocols, and ports so each rule is applied to the origin it can actually control.

By Robots.txt Tools Editorial TeamLast verified 2026-09-133 min read

Editorial method: this guide documents the review workflow used by the Robots.txt Tools Editorial Team. Procedural first-person wording describes that method; it is not a claim of client work or personal case history.

Robots.txt for Subdomains: How We Check Scope Before Changing Rules

When crawler behavior looks wrong, we check scope before syntax. A valid rule can still have no effect when it is served from the wrong origin.

We start with protocol, host, and port

Robots.txt is origin-scoped. These URLs do not automatically share one policy:

https://example.com/robots.txt
https://www.example.com/robots.txt
https://shop.example.com/robots.txt
http://example.com/robots.txt
https://example.com:8443/robots.txt

That means a root-domain file does not cascade to shop.example.com, and an HTTPS file is not automatically the policy for an independently reachable HTTP origin.

We inventory every public host that matters

Before editing, we list production, www, shop, docs, API-facing web hosts, previews, and staging environments that search engines can reach. Then we fetch /robots.txt from each origin. This often exposes the real problem: one host serves a stale file, another returns 404, while a third is controlled by a different platform.

We identify deployment ownership

A multi-host site may combine Next.js, Shopify, WordPress, a documentation platform, and a CDN. We therefore ask which system actually owns the live response. Editing a repository file cannot change a robots.txt generated by a CMS, reverse proxy, edge function, or platform setting.

After deployment, our robots.txt checker keeps the review focused on the public response rather than the file we expected to be public.

We test URLs against the correct host

If the target is:

https://shop.example.com/checkout/

we test it against the policy served by shop.example.com, not a similar path on the apex domain. For overlapping rules, we use the robots.txt tester and review the winning rule rather than relying on line order.

We treat staging and preview hosts separately

Preview hosts do not inherit production robots.txt. If an environment must be private, we use authentication or network access control. Disallow: / is crawler guidance, not security. If a public staging URL should stay out of search, we also evaluate noindex and removal workflows instead of assuming a crawl block solves indexing.

We verify migration edges

During host or protocol migrations, we check both old and new origins. www and non-www, HTTP and HTTPS, or legacy subdomains may still receive crawler requests from old links and redirects. Robots.txt is not a redirect mechanism, so canonical host migration belongs at the HTTP routing layer.

Our scope checklist

  1. list every public protocol + host + port;
  2. fetch /robots.txt from each origin;
  3. identify the deployment owner of each response;
  4. check old HTTP and redirect hosts;
  5. inspect important subdomains independently;
  6. test representative URLs on their own host;
  7. use authentication for private environments;
  8. verify the live response after every deployment.

The key model is simple: robots.txt belongs to an origin, not to an organization. Once we map the actual origins first, subdomain problems become much easier to diagnose without adding unnecessary rules.

Sources

Related guides