Primary-source first
Protocol behavior is checked against RFC 9309. Search-engine-specific behavior is checked against that provider's official documentation. Crawler presets for services such as OpenAI, Anthropic, Perplexity, and Google are not inferred from third-party SEO tool lists when a primary source is available.
Protocol rules and provider behavior are kept separate
The Robots Exclusion Protocol defines general parsing and matching behavior, while individual providers may document additional crawler behavior or unsupported directives. The validator therefore avoids presenting every compatibility warning as an RFC syntax error.
Examples are starting points, not universal prescriptions
WordPress, Blogger, staging, and other presets are editable examples. A path that is safe to block on one site can be important on another, so presets are intentionally presented with review warnings instead of being described as universally “best” configurations.
Material crawler changes are re-verified
AI search and crawler policies can change. When a provider changes a crawler name, purpose, robots.txt behavior, or documentation, the affected preset and explanatory copy should be reviewed against the new primary source before the change is published.
Corrections
Reproducible parser bugs, outdated crawler descriptions, and source corrections should be treated as product defects and fixed in the shared rules engine or source metadata rather than patched only in one page's prose.