In Webflow, we treat robots.txt as part of the publishing workflow. The relevant controls live under Site settings → SEO → Indexing, but only the published response matters to crawlers.
We inspect production before editing
We open https://example.com/robots.txt and compare it with Webflow settings. We look for broad Disallow rules, a sitemap reference using the wrong domain, and duplicate Sitemap: lines.
We check the automatic sitemap behavior
Webflow normally adds the site sitemap reference. Before adding another line manually, we verify whether it is already present. In reverse-proxy or multi-site architectures, Webflow also documents options for removing the automatic sitemap reference so a different root-domain sitemap can be used; plan and architecture requirements still need to be checked.
Save is not enough: we publish
Our sequence is edit → save → publish → fetch /robots.txt from the production domain. If the public response is still old, we investigate publishing, domain routing, and cache before rewriting the rules again.
Content-Signal is a separate mechanism
Webflow exposes a Content-Signal HTTP header for AI content-usage preferences. We keep that concept separate from robots.txt. A crawler-access decision and a provider/content-usage preference are not the same control and should not be mixed in one policy model.
We test bot-specific groups carefully
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /
A configuration like this can block many legitimate crawlers. We validate it and test representative URLs with the robots.txt tester before release.
Private previews still need access control
Robots.txt is public. Confidential Webflow previews need authentication or another real access-control layer. If a public page should stay out of search, we review page indexing/noindex controls rather than assuming a crawl block is equivalent.
Our Webflow release checklist
- open the production file;
- review SEO → Indexing;
- save and publish;
- avoid duplicate sitemap references;
- verify the canonical domain;
- test crawler-specific groups;
- keep Content-Signal separate from robots.txt;
- protect private previews with real access control;
- inspect the live file with the checker.
We change only rules we can justify and always verify the response Webflow actually publishes.
