Query parameters can multiply a small path set into thousands of crawlable URLs, so we do not start with a blanket Disallow. We first decide whether the real problem is crawl waste, duplicate content, indexation, tracking noise, or an application that generates unnecessary URLs.
We classify parameters by purpose
Typical examples include:
/products?sort=price
/products?color=black
/search?q=shoes
/article?utm_source=newsletter
Sorting, filters, faceted navigation, internal search, tracking, sessions, and pagination have different SEO roles. A rule that is safe for ?utm_source= may be harmful for a valuable filter state.
We test the exact query pattern
A targeted pattern can be useful:
User-agent: *
Disallow: /*?sort=
But we never approve it by eye alone. We test representative URLs, including allowed exceptions, with the robots.txt tester. Parameter position, encoding, and overlapping path rules can change the result.
Duplicate does not automatically mean block
For tracking variants such as /article?utm_source=newsletter, canonicalization may already be the cleaner control. Blocking the URL can stop a crawler from fetching page-level signals. We therefore ask, in order: can the application stop generating the URL, is a redirect appropriate, does canonical solve duplication, does the page need noindex, and only then whether crawl blocking is still useful.
Faceted navigation needs an allowlist mindset
Ecommerce filters can create both valuable landing pages and near-infinite combinations. We avoid Disallow: /*? unless crawl evidence shows that every query-string URL is intentionally disposable. Where selected facets deserve indexing, we make that architecture explicit rather than hoping a broad wildcard will distinguish useful combinations.
Google now documents faceted-navigation crawl control directly
Google's current crawling documentation explicitly calls out faceted navigation as a source of effectively unbounded URL spaces. When filtered URLs do not need to appear in search, Google recommends preventing crawling of those parameter patterns or using URL fragments for filters. When filtered URLs do need to be crawlable, the architecture should keep parameter order predictable, avoid duplicate or nonsensical combinations, and return a real 404 for empty combinations rather than redirecting all of them to one generic page.
A targeted pattern can therefore be appropriate:
User-agent: Googlebot
Disallow: /*?*color=
Disallow: /*?*size=
Allow: /*?products=all$
But we only use patterns like these after confirming that valuable landing pages are not hidden inside the same parameter space.
We separate four common parameter jobs
| Parameter type | Typical first choice |
|---|---|
Tracking such as utm_source | Canonicalization / URL hygiene |
Internal search such as ?q= | Usually keep out of crawl/index surfaces |
Sorting such as ?sort=price | Often crawl reduction if no search value |
| Facets such as color/size/category | Decide which combinations deserve indexable landing pages |
This prevents a frequent mistake: using one wildcard to solve crawl waste, duplication, indexing, and analytics noise at the same time. Those are different problems and can require different controls.
We separate crawling from indexing
If the requirement is “do not show this URL in search,” robots.txt alone may be the wrong layer. A blocked URL can still be discovered. Public URLs that must not be indexed generally need noindex while remaining crawlable; duplicates may need canonical or redirects; private data needs authentication. Our robots.txt vs meta robots guide covers those layers.
We check whether links are creating the crawl problem
Robots.txt can stop compliant crawlers from fetching a parameter pattern, but it does not remove the links that generate that URL space. We therefore inspect navigation, filter widgets, pagination, JavaScript routing, and XML sitemaps for unnecessary parameter URLs. If the application can stop exposing disposable combinations, that usually gives us a cleaner long-term result than expanding the robots file forever.
We also sample server logs when available. A large number of crawler requests to low-value parameter combinations is evidence for crawl-control work; a small number of requests with mostly duplicate indexing issues may point us back toward canonicalization or application cleanup instead.
We use a test matrix before deployment
We write expected outcomes before changing the file:
| URL | Expected |
|---|---|
/products | Allow |
/products?sort=price | Disallow |
/products?color=black | Allow |
/products?page=2 | Allow |
/search?q=boots | Disallow |
Then we run the same matrix again after publishing and fetch the live file with the robots.txt checker. This catches stale CDN output and deployment mismatches.
Our parameter checklist
We confirm what each parameter changes, whether crawlable links generate it, whether any combinations have search value, whether canonical/noindex/redirects solve the problem better, whether the wildcard has unwanted matches, and whether production behaves like the draft.
Our default is not “query strings are bad.” We reduce unnecessary URL generation first and use robots.txt only for the part of the problem that is genuinely about crawler access.
