When a robots.txt edit appears to be ignored, we separate three different states: what the repository contains, what production serves now, and what a crawler may still have cached.
We check production before blaming crawler cache
We open the exact origin /robots.txt and inspect the current response. A wrong deployment, stale CDN object, host mismatch, redirect, or server error can look like “Google has not refreshed robots.txt” even when the crawler is not the problem.
Our robots.txt checker is designed for this live-response step.
We account for normal caching
Google documents that robots.txt is generally cached for up to about 24 hours, although cache behavior can vary with availability and response headers. We therefore do not repeatedly republish the file every few minutes just because crawler behavior has not changed immediately.
We compare the exact host
Cache debugging is origin-specific. www.example.com, example.com, and shop.example.com can serve different files. A successful fetch on one host does not prove another crawler-facing origin was updated.
We inspect delivery layers
If the live response is old, we trace the path from application to CDN and edge cache. We check build output, routing rules, reverse proxies, cache invalidation, and whether a platform generates robots.txt dynamically. A repository commit is not proof of public delivery.
Google's documented cache behavior gives us a baseline
Google currently says robots.txt content is generally cached for up to about 24 hours. The cached version can live longer when Google cannot refresh it because of timeouts or 5xx responses. Google also notes that cached robots responses may be shared by multiple crawlers and that cache-control headers such as max-age can influence how long the file is cached.
That gives us a more useful timeline than repeatedly editing the file:
- Production still serves old text: fix deployment/CDN ownership first.
- Production serves new text but crawler behavior is unchanged for a few hours: normal cache is plausible.
- Production returns intermittent 5xx/timeouts: refresh may be delayed; fix availability.
- More than a normal cache window has passed: verify the exact origin, logs, Search Console evidence, and whether a different crawler/user-agent is involved.
Cache headers are not a substitute for a correct deployment
We inspect Cache-Control, CDN rules, and edge configuration, but we do not try to "force Google" by publishing extreme cache directives. The first requirement is a stable, fetchable /robots.txt response. A low TTL does not help if a reverse proxy still serves the wrong file or if the origin fails during crawler refreshes.
We keep a before/after evidence set
For risky changes, we save the previous robots.txt text, deployment time, HTTP status, final URL, response headers, and the new body. That evidence lets us tell the difference between a crawler cache, a CDN cache, and an accidental rollback without making another speculative edit.
Error responses change the diagnosis
A temporary 5xx or rate-limited response is different from a healthy 200 containing old text. We inspect HTTP status before reasoning about the rules themselves. The HTTP status guide covers crawler behavior when robots.txt cannot be fetched normally.
A simple diagnostic table keeps cache incidents short
| What we observe | Most likely next check |
|---|---|
| Browser and checker both show old text | Deployment/CDN ownership |
| Live file is new, crawler behavior is old | Normal crawler cache or recrawl delay |
| Different hosts show different versions | Origin-specific configuration |
| Intermittent 5xx/timeouts | Availability before cache |
| Rules are new but outcome is unexpected | Matching logic, not cache |
We use this table before touching the file again. A second robots.txt edit while the first one is still propagating can make the timeline harder to reconstruct and can introduce a new crawl policy before we understand the first result.
Our verification sequence
- fetch production
/robots.txtfrom the exact host; - record HTTP status, final URL, and response body;
- compare it with the intended deployment;
- check CDN or platform caches;
- allow for normal crawler cache, commonly around 24 hours;
- validate the live file;
- test representative URLs after the expected refresh window;
- use Search Console or logs when available to distinguish crawler state from delivery state.
We make another edit only when the current file is actually wrong. That prevents a simple cache delay from turning into a sequence of unnecessary rule changes.
