Copyable patterns

Robots.txt Examples

Common robots.txt patterns for Allow, Disallow, Sitemap, wildcards, and crawler-specific rules. Treat them as starting points and verify the result for your own site before publishing.

Allow all crawlers

An empty Disallow value does not block any path. A robots.txt file is not required if the site has no crawl restrictions.

User-agent: *
Disallow:

Block all crawlers

Blocks compliant crawlers from the entire site. Use carefully and do not rely on this as access control.

User-agent: *
Disallow: /

Block one directory

Blocks URLs whose path starts with /private/.

User-agent: *
Disallow: /private/

Allow inside a blocked directory

A more specific Allow rule can permit one path inside a broader blocked area.

User-agent: *
Disallow: /admin/
Allow: /admin/help/

Add a sitemap

Sitemap directives use an absolute HTTP(S) URL.

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Block a file pattern

The * wildcard can match a sequence of characters in a path pattern.

User-agent: *
Disallow: /*.pdf$

Block Googlebot only

A specific Googlebot group is evaluated separately from the wildcard group.

User-agent: Googlebot
Disallow: /private-for-google/

Allow Googlebot while blocking the wildcard group

A specific Googlebot group takes precedence over the wildcard fallback for Googlebot.

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

Block GPTBot but allow OAI-SearchBot

OpenAI documents GPTBot and OAI-SearchBot as separate controls. Review current provider documentation before publishing crawler-specific rules.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Site type

Start with a suggested path or add your own. Only block paths that you intentionally want compliant crawlers to avoid.

WordPress

Starts with /wp-admin/ blocked while keeping admin-ajax.php crawlable.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

E-commerce

Starts open, with common cart, checkout, account and search paths available as quick choices.

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

Staging / block all

Blocks compliant crawlers from the entire site. Use authentication too when the environment must stay private.

User-agent: *
Disallow: /

Staging mode intentionally blocks the whole site. Use authentication when the environment must stay private.

Build your own file

Use the generator when you need to combine several paths, add multiple sitemaps, or create crawler-specific groups.

Open Robots.txt Generator

Test before publishing

Overlapping rules and specific User-agent groups can change the result. Test representative URLs instead of assuming an example behaves the same on every site.

Open Robots.txt Tester →