Large robots.txt files usually signal an architecture problem before they become a parser problem. We check both file size and encoding because a file can look normal in an editor while the delivered bytes are too large or contain characters a crawler cannot interpret as intended.
We measure the delivered file, not the source template
Google documents a 500 KiB robots.txt size limit. Content beyond the supported limit may be ignored, so we measure the production response rather than a pre-render template. Generated comments, repeated user-agent groups, or platform output can make the live file larger than the source suggests.
We keep UTF-8 predictable
Robots.txt should be served as UTF-8. We avoid smart-copy artifacts, legacy encodings, and hidden invalid byte sequences. Non-ASCII paths can be valid, but the bytes and URL escaping still need to match what the crawler evaluates.
When debugging an encoding issue, we compare the raw response with the URLs being tested instead of assuming the text displayed by a browser is byte-for-byte identical to the crawler input.
What happens beyond 500 KiB
Google currently enforces a 500 KiB robots.txt size limit and ignores content beyond that supported size. That makes line ordering operationally important in an oversized file: a rule that exists in the repository but falls after the processed byte limit may effectively not exist for Google.
We therefore do not aim for "499 KiB and technically valid." We leave a large safety margin so comments, generated groups, platform additions, and future crawler rules cannot push critical directives beyond the processed range.
Encoding bugs often appear as matching bugs
A file can render legibly in a browser while still containing bytes or percent-encoding differences that change matching. We check:
- UTF-8 output from the server rather than the editor's local encoding;
- whether non-ASCII paths are represented consistently;
- whether a UTF-8 BOM or replacement character was introduced by a generator;
- whether path case matches the real URL;
- whether the delivered response is actually text rather than an HTML error page.
For a path containing non-ASCII characters, we test both the visible URL and the percent-encoded form in our matching tool. The goal is not to manually encode every rule; it is to make sure the rule and the URL normalize to the same intended resource.
Oversized files usually deserve architecture cleanup
Thousands of per-product or per-query exclusions are difficult to reason about and often indicate that URL generation should be fixed closer to the application. We first look for stable directory or parameter patterns, remove duplicate groups, and question giant third-party bot lists. A shorter file is easier to audit, less likely to cross crawler limits, and less likely to hide a dangerous whole-site rule.
We reduce duplication before compressing logic
If the file approaches 500 KiB, we look for repeated rules, generated per-URL blocks, and bot lists that could be expressed more simply. Robots.txt is not the right place to enumerate thousands of individual URLs when a stable path rule can represent the same crawl policy.
We also avoid “optimization” that makes the file impossible to maintain. A shorter rule set is useful only when its matching behavior remains understandable and testable.
We validate after every structural change
We run the final text through the robots.txt validator and test representative URLs with the robots.txt tester. File-size cleanup can accidentally change group boundaries or wildcard scope if lines are merged carelessly.
HTTP delivery still matters
A small, valid UTF-8 file can still fail if production serves the wrong content type, redirects unexpectedly, returns an error, or exposes a stale CDN version. We inspect the live response with the robots.txt checker after deployment.
Our size and encoding checklist
- measure the actual production response in bytes;
- keep the file comfortably below 500 KiB;
- serve predictable UTF-8 text;
- remove repeated or generated per-URL rules where a stable pattern works;
- preserve clear user-agent group boundaries;
- validate syntax after simplification;
- test real URLs for the affected crawlers;
- verify the live response, not only repository content.
The safest robots.txt file is usually small enough to review, encoded consistently, and built from a few deliberate rules rather than a growing list of exceptions.
