We choose controls by the problem they solve. Robots.txt manages crawler access; noindex manages indexing when the crawler can read it; canonical consolidates duplicate signals; redirects move users and crawlers; authentication protects private content.
Robots.txt answers a crawl question
User-agent: *
Disallow: /internal-search/
This can reduce crawler access to a path, but it is not a reliable request to remove a URL from search results. A URL may still be discovered through links or historical signals.
Meta robots answers an indexing question
For public HTML that can be crawled but should not appear in search, we use:
<meta name="robots" content="noindex">
The crawler must be able to fetch the page to see the directive. Blocking the same URL in robots.txt can prevent that processing.
X-Robots-Tag handles response-level indexing
For PDFs and other non-HTML resources, or when headers are easier to manage centrally, X-Robots-Tag can carry noindex in the HTTP response:
X-Robots-Tag: noindex
Again, the crawler needs access to the response.
Canonical is not a removal command
A canonical link is a consolidation signal for duplicate or near-duplicate pages. We do not use it as a privacy control or as a guaranteed substitute for noindex. If an obsolete URL should permanently move, a redirect may be clearer.
Authentication is the privacy boundary
If content must be private, we use authentication, authorization, or network restrictions. Robots.txt is publicly readable, and noindex still assumes the resource is accessible to a crawler. Neither is a security mechanism.
Common combinations and what they actually achieve
| Configuration | Crawl? | Indexing intent |
|---|---|---|
robots.txt Disallow only | Blocked for compliant crawler | URL can still be known or appear without content |
noindex meta only | Allowed | Request exclusion from search |
X-Robots-Tag: noindex | Allowed | Request exclusion, including non-HTML resources |
Disallow + page noindex | Crawler may not fetch page | The noindex may not be seen |
| Authentication | Access denied | Stronger privacy boundary than crawler directives |
This table is why we avoid advice like "block it in robots.txt and add noindex for extra safety." Those two controls can work against each other when the crawler must fetch the page to discover the indexing directive.
When X-Robots-Tag is the cleaner option
For PDFs, generated files, feeds, or other resources without a convenient HTML <head>, an HTTP header can be easier to enforce:
X-Robots-Tag: noindex
We verify the header on the final 200 response, not only in server configuration. Redirects, CDN behavior, and storage layers can alter headers before the crawler receives them.
When robots.txt is still the right tool
Robots.txt is appropriate when the actual goal is to reduce crawler access to low-value URL spaces, such as internal search or an unbounded filter system, and we accept the indexing implications. It is also useful for crawler-specific access preferences. It is not our first choice for removing a public URL from results, consolidating duplicates, or protecting private information.
We use a decision sequence
- private or sensitive → authentication;
- public but must stay out of search →
noindexorX-Robots-Tagwhile crawlable; - duplicate content → canonical, redirects, or URL cleanup;
- crawl waste → consider robots.txt after checking indexing/rendering side effects;
- crawler-specific policy → a targeted robots.txt user-agent group.
For a deeper indexing example, see robots.txt vs noindex.
We diagnose by the symptom, not by the directive name
When someone says "Google should not see this page," we clarify what they mean before changing anything. "Do not spend crawl resources here," "do not show this URL in search," "merge this duplicate into another URL," and "nobody should access this page" are four different requirements.
That clarification prevents a common escalation pattern where teams add Disallow, then add noindex, then add a canonical, and finally add a redirect without knowing which signal is supposed to win. We prefer one primary control for the actual goal, then only add complementary controls whose interaction we can explain and verify.
We verify both layers after deployment
We fetch the live robots.txt with the checker, inspect the rendered HTML or HTTP headers, and test representative URLs with the tester. A correct source file is not enough if a CDN, framework, or CMS serves something different.
Keeping these controls separate prevents the common failure where robots.txt blocks the crawler that was supposed to read a noindex instruction.
