WordPress crawler control
WordPress Robots.txt Generator
Start from a conservative WordPress-oriented preset, then customize paths, crawler groups, and your sitemap for the site you actually run. The goal is not to copy a universal “best robots.txt,” but to create rules you can explain and test.
Choose your editing mode
Simple mode is safer for quick setup. Advanced mode exposes every crawler group and rule.
Crawler purpose policy
Start neutral, then opt in to AI crawler restrictions only when they match your policy.
Neutral default: keep search, AI search and AI-control crawlers open unless your path rules explicitly block them.
Paths to block
Start with a suggested path or add your own. Only block paths that you intentionally want compliant crawlers to avoid.
Suggested paths
Add custom path
Add any site path. Bare paths get a leading slash; full HTTP(S) URLs are reduced to their path and query.
Search
Allowed by default
AI Search
Allowed by default
AI Control / Training
Allowed by default
Before you publish
- 1. Generate
Review the live output. - 2. Validate
Check syntax and compatibility. - 3. Test URLs
Verify important paths and crawlers. - 4. Publish
Upload it as /robots.txt at your site root.
Generated robots.txt
Live output; generation and validation stay in your browser.
User-agent: *Allow: /wp-admin/admin-ajax.phpDisallow: /wp-admin/ No syntax or compatibility warnings detected.
What the WordPress preset does
WordPress core itself can generate robots.txt output dynamically. Its core robots response includes an admin-directory Disallow rule and an Allow rule for the admin AJAX endpoint. This preset follows that conservative pattern as a starting point, while leaving the rest of your site open unless you add more rules.
Disallow /wp-admin/
Keeps compliant crawlers out of the WordPress administration directory. This is crawler guidance, not a security boundary; the admin area still needs normal authentication and hardening.
Allow /wp-admin/admin-ajax.php
Preserves crawl access to a front-end endpoint that themes and plugins may rely on. A broad /wp-admin/ rule should not accidentally hide a resource that public pages need.
Primary WordPress reference: WordPress Core do_robots() documentation.
WordPress robots.txt decisions worth reviewing
Plugins, ecommerce features, faceted navigation, internal search, and custom post types can create URLs that do not exist on a basic WordPress install. Review those URL patterns before adding broad Disallow rules.
Internal search
If you block WordPress search-result URLs, confirm that the rule matches only the search paths or parameters you intend. Do not use a broad pattern that also catches normal posts or category pages.
WooCommerce and account flows
Cart, checkout, account, wishlist, and filtered catalog URLs may be candidates for crawl control, but their exact paths vary by configuration and plugins. Test representative product and category URLs before publishing.
Faceted and parameter URLs
Filters can create many crawlable URL combinations. robots.txt can reduce crawling of selected patterns, but it is not a substitute for a deliberate canonical and indexing strategy.
XML sitemaps
Use the sitemap URL your production site actually serves. WordPress core or an SEO plugin may provide the sitemap, so verify the live URL instead of assuming every WordPress site uses the same endpoint.
Virtual vs. physical robots.txt on WordPress
WordPress can serve robots.txt dynamically when the request reaches WordPress. A server-level or physical robots.txt file can change what is actually returned. Plugins can also alter the generated response.
For SEO, the important file is the one crawlers receive from the production URL. After publishing any change, use the live Checker instead of assuming the WordPress editor, plugin setting, or file on disk is the final response.
Before you publish a WordPress robots.txt change
- 1
Confirm the production paths
Check actual admin, search, cart, account, filter, and sitemap URLs.
- 2
Validate the full file
Catch malformed groups, bad sitemap directives, and risky whole-site blocks.
- 3
Test representative URLs
Include the home page, a post, a page, a category, a product if applicable, and every path you intend to block.
- 4
Check the live response
After deployment, fetch /robots.txt from the public host and confirm the expected rules are actually being served.
WordPress robots.txt FAQ
Does every WordPress site need a custom robots.txt file?
No. WordPress can generate robots.txt behavior dynamically, and a custom file should be added only when you have a specific crawl policy to implement and test.
Should I block wp-admin?
WordPress core includes an admin-directory Disallow rule while allowing the admin AJAX endpoint. That is crawler guidance only; it does not secure the admin area.
Should I block WordPress search and filter pages?
Sometimes, but there is no universal rule. Decide from your actual URL patterns and crawl goals, then test important pages so a broad pattern does not block useful content.
Can robots.txt noindex WordPress pages?
No. robots.txt primarily controls crawling. If the goal is to keep a crawlable URL out of search results, use an appropriate indexing control rather than relying on Disallow alone.