Explainable rule testing
Robots.txt Tester
Paste a robots.txt file, choose a crawler, and test a URL. The result explains the matching User-agent group and the most specific Allow or Disallow rule.
robots.txt rules
Paste rules, load a public file, or send rules here directly from Generator.
Test result
Decisions use the same matching engine as Checker and Generator diagnostics.
Blocked
https://example.com/admin/login
Disallow rule on line 2 is the most specific matching rule.
Decision trace
- Crawler group
- * (wildcard fallback)
- Winning rule
- Disallow: /admin/
- Source line
- 2
- Rule specificity
- 7 normalized bytes
Crawler comparison
Compare the first tested URL across search, AI search and AI-control crawler tokens: https://example.com/admin/login
Googlebot
Search
Disallow: /admin/
Bingbot
Search
Disallow: /admin/
OAI-SearchBot
AI Search
Disallow: /admin/
GPTBot
AI Control / Training
Disallow: /admin/
Claude-SearchBot
AI Search
Disallow: /admin/
ClaudeBot
AI Control / Training
Disallow: /admin/
PerplexityBot
AI Search
Disallow: /admin/
Google-Extended
AI Control / Training
Disallow: /admin/
Why the winning rule matters
robots.txt matching is not simply “first rule wins.” A crawler selects the applicable User-agent group and then evaluates matching Allow and Disallow patterns. This tester surfaces the rule used for its decision instead of returning only a yes/no result.
Validate the whole file →Decision trace
Decisions use the same matching engine as Checker and Generator diagnostics.
Crawler comparison
Parseable rules are still evaluated when only warnings exist.
Bulk URLs
Up to 100 unique non-empty lines per run.
Technical references
Rules and crawler-specific guidance are checked against primary standards and provider documentation rather than copied from other SEO tools.
- RFC 9309
Robots Exclusion Protocol syntax, matching, grouping, and percent-encoding behavior.
- Google robots.txt specification
Google-specific robots.txt behavior and supported fields.