For Magento / Adobe Commerce, we first identify which layer owns the storefront response. Native Commerce, Cloud infrastructure, a reverse proxy, and a headless/PWA frontend can place robots.txt responsibility in different systems.
We start in Content → Design → Configuration
Adobe Commerce exposes Search Engine Robots settings under Content → Design → Configuration. We review the configuration for the relevant store view and confirm what the public /robots.txt actually serves.
Default Robots is not the same as the raw crawl file
The Default Robots setting controls page-level robot behavior such as index/follow defaults. We do not confuse that with custom robots.txt crawl rules. Indexing directives and crawler access are different layers.
We preserve platform-critical paths deliberately
Ecommerce sites have carts, customer areas, layered navigation, search, media, APIs, and generated routes. We do not paste a generic Magento blocklist. We map the real storefront URLs, decide which ones have search value, and test the exact patterns.
Cloud environments need environment-aware indexing controls
Adobe Commerce Cloud adds deployment and environment considerations. Staging or integration environments should use real access controls where possible and should not inherit a production crawl/index policy accidentally. Sitemap paths and generated files also need to match the environment that search engines can reach.
Headless and PWA architectures change ownership
If Magento is the commerce backend but a PWA or headless frontend serves the public hostname, the frontend/CDN may own /robots.txt. Editing the Commerce Admin setting is not enough unless that setting actually controls the public response. We verify the origin and delivery chain before changing rules.
We test faceted ecommerce URLs
Magento stores can generate large numbers of filter and parameter combinations. Before blocking them, we decide which facets deserve indexing, which should canonicalize, and which are pure crawl expansion. Our query-parameter guide covers that decision.
Our Adobe Commerce checklist
- identify the public storefront architecture;
- review Content → Design → Configuration → Search Engine Robots;
- distinguish Default Robots from robots.txt crawl rules;
- inspect Cloud environment/indexing settings where applicable;
- verify sitemap references and canonical hostnames;
- test filters, search, cart, account, and other representative URLs;
- confirm whether PWA/headless/CDN owns the public response;
- validate and inspect production with the robots.txt checker.
The key is ownership: we configure the layer that actually serves the public robots.txt response, then verify ecommerce-specific crawl decisions with real URLs rather than relying on a generic platform template.
