guides
Robots.txt for AI Crawlers: Search, Training and Bot Access
How should robots.txt handle AI crawlers without accidentally blocking search discovery?
Read guide →Knowledge base
Robots.txt Guides
Practical guides for crawler rules, indexing controls, host scope, parameter URLs, deployment behavior, and robots.txt troubleshooting.
Robots.txt Guides guides
guides
Robots.txt vs Noindex: Crawling and Indexing Explained
Can robots.txt prevent a page from being indexed, and when should noindex be used instead?
Read guide →
guides
Robots.txt Crawl-delay: What Google Supports and What It Does Not
Does crawl-delay work in robots.txt, especially for Googlebot?
Read guide →
guides
Robots.txt Wildcards and Rule Precedence: *, $ and Longest Match
How do robots.txt wildcards and conflicting Allow or Disallow rules actually match?
Read guide →
guides
Robots.txt HTTP Status Codes: 200, 404, 403, 429 and 5xx
What happens when robots.txt returns 404, 403, 429, 500, or another HTTP status?
Read guide →
guides
Sitemap in Robots.txt: How we Add and Verify the Directive
Where should we add a Sitemap directive in robots.txt, and how do we verify multiple sitemap references?
Read guide →
guides
Robots.txt for Subdomains: How we Check Scope Before Changing Rules
Does the root-domain robots.txt apply to subdomains, HTTP, HTTPS, or another port?
Read guide →
guides
Robots.txt Query Parameters: How we Test Faceted and Parameter URLs
How should we control query-parameter and faceted URLs without blocking valuable pages by accident?
Read guide →
guides
Robots.txt File Size and Encoding: How we Debug Large or Broken Files
What happens when robots.txt is too large or not encoded as UTF-8, and how do we verify the live response?
Read guide →
guides
Robots.txt vs Meta Robots: How we Choose the Right Control Layer
Should we use robots.txt, a meta robots tag, X-Robots-Tag, canonical, or authentication for this URL?
Read guide →
guides
Robots.txt Cache and Updates: How we Verify a Change Really Took Effect
How long can robots.txt stay cached, and how do we tell a crawler cache from a failed production deployment?
Read guide →
guides
Staging Site Robots.txt: How we Keep Test Environments Out of Search
How should we keep a staging or preview site out of search without treating robots.txt as a security control?
Read guide →