Query parameters can multiply a small path set into thousands of crawlable URLs, so we do not start with a blanket Disallow. We first decide whether the real problem is crawl waste, duplicate content, indexation, tracking noise, or an application that generates unnecessary URLs.
We classify parameters by purpose
Typical examples include:
/products?sort=price
/products?color=black
/search?q=shoes
/article?utm_source=newsletter
Sorting, filters, faceted navigation, internal search, tracking, sessions, and pagination have different SEO roles. A rule that is safe for ?utm_source= may be harmful for a valuable filter state.
We test the exact query pattern
A targeted pattern can be useful:
User-agent: *
Disallow: /*?sort=
But we never approve it by eye alone. We test representative URLs, including allowed exceptions, with the robots.txt tester. Parameter position, encoding, and overlapping path rules can change the result.
Duplicate does not automatically mean block
For tracking variants such as /article?utm_source=newsletter, canonicalization may already be the cleaner control. Blocking the URL can stop a crawler from fetching page-level signals. We therefore ask, in order: can the application stop generating the URL, is a redirect appropriate, does canonical solve duplication, does the page need noindex, and only then whether crawl blocking is still useful.
Faceted navigation needs an allowlist mindset
Ecommerce filters can create both valuable landing pages and near-infinite combinations. We avoid Disallow: /*? unless crawl evidence shows that every query-string URL is intentionally disposable. Where selected facets deserve indexing, we make that architecture explicit rather than hoping a broad wildcard will distinguish useful combinations.
We separate crawling from indexing
If the requirement is “do not show this URL in search,” robots.txt alone may be the wrong layer. A blocked URL can still be discovered. Public URLs that must not be indexed generally need noindex while remaining crawlable; duplicates may need canonical or redirects; private data needs authentication. Our robots.txt vs meta robots guide covers those layers.
We use a test matrix before deployment
We write expected outcomes before changing the file:
| URL | Expected |
|---|---|
/products | Allow |
/products?sort=price | Disallow |
/products?color=black | Allow |
/products?page=2 | Allow |
/search?q=boots | Disallow |
Then we run the same matrix again after publishing and fetch the live file with the robots.txt checker. This catches stale CDN output and deployment mismatches.
Our parameter checklist
We confirm what each parameter changes, whether crawlable links generate it, whether any combinations have search value, whether canonical/noindex/redirects solve the problem better, whether the wildcard has unwanted matches, and whether production behaves like the draft.
Our default is not “query strings are bad.” We reduce unnecessary URL generation first and use robots.txt only for the part of the problem that is genuinely about crawler access.
