Live website audit
Robots.txt Checker
Enter a website URL to fetch its root robots.txt file and inspect the HTTP response, file size, and validation warnings.
Fetches the site-root /robots.txt, safely follows up to five redirects, blocks private/local destinations, and stops reading above 500 KiB.
Enter a public website to check whether robots.txt exists, whether search crawlers can access the root, and what needs attention.
Checker vs. validator
The checker fetches the live /robots.txt file from a website. The validator does not make a network request; it analyzes text you paste directly.
Need to test one URL?
Use the tester when your question is whether a specific URL is allowed or blocked for a particular crawler.
Open Robots.txt Tester →Quick result
Start here. Expand Technical details only when you need HTTP, redirect, crawler-group, or validation information.
Search root access
Search root access checks only /. Use Tester to verify important URLs and paths individually or in bulk.
What should I fix?
Prioritized from crawler access and validation diagnostics.
Technical references
Rules and crawler-specific guidance are checked against primary standards and provider documentation rather than copied from other SEO tools.
- RFC 9309
Robots Exclusion Protocol syntax, matching, grouping, and percent-encoding behavior.
- Google robots.txt specification
Google-specific robots.txt behavior and supported fields.