Robots.txt Guides

Robots.txt Cache: How We Verify That a Change Is Really Live

Our workflow for separating crawler cache from deployment, CDN, redirect, and HTTP-status problems when a robots.txt change appears not to work.

By Robots.txt Tools Editorial TeamLast verified 2026-09-132 min read

Editorial method: this guide documents the review workflow used by the Robots.txt Tools Editorial Team. Procedural first-person wording describes that method; it is not a claim of client work or personal case history.

Robots.txt Cache: How We Verify That a Change Is Really Live

When a robots.txt edit appears to be ignored, we separate three different states: what the repository contains, what production serves now, and what a crawler may still have cached.

We check production before blaming crawler cache

We open the exact origin /robots.txt and inspect the current response. A wrong deployment, stale CDN object, host mismatch, redirect, or server error can look like “Google has not refreshed robots.txt” even when the crawler is not the problem.

Our robots.txt checker is designed for this live-response step.

We account for normal caching

Google documents that robots.txt is generally cached for up to about 24 hours, although cache behavior can vary with availability and response headers. We therefore do not repeatedly republish the file every few minutes just because crawler behavior has not changed immediately.

We compare the exact host

Cache debugging is origin-specific. www.example.com, example.com, and shop.example.com can serve different files. A successful fetch on one host does not prove another crawler-facing origin was updated.

We inspect delivery layers

If the live response is old, we trace the path from application to CDN and edge cache. We check build output, routing rules, reverse proxies, cache invalidation, and whether a platform generates robots.txt dynamically. A repository commit is not proof of public delivery.

Error responses change the diagnosis

A temporary 5xx or rate-limited response is different from a healthy 200 containing old text. We inspect HTTP status before reasoning about the rules themselves. The HTTP status guide covers crawler behavior when robots.txt cannot be fetched normally.

Our verification sequence

  1. fetch production /robots.txt from the exact host;
  2. record HTTP status, final URL, and response body;
  3. compare it with the intended deployment;
  4. check CDN or platform caches;
  5. allow for normal crawler cache, commonly around 24 hours;
  6. validate the live file;
  7. test representative URLs after the expected refresh window;
  8. use Search Console or logs when available to distinguish crawler state from delivery state.

We make another edit only when the current file is actually wrong. That prevents a simple cache delay from turning into a sequence of unnecessary rule changes.

Related guides