Next.js Robots.txt: How We Use App Router Metadata Safely

With the Next.js App Router, we choose between a static app/robots.txt file and a generated app/robots.ts or app/robots.js metadata route. We make that choice based on whether the policy is truly static or depends on deployment configuration.

Static policy: app/robots.txt

For a small site with one stable policy, a plain file is easy to audit:

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

The advantage is predictability. There is very little runtime logic to fail.

Generated policy: app/robots.ts

When environment-aware output is justified, Next.js supports MetadataRoute.Robots:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/' },
    sitemap: 'https://example.com/sitemap.xml',
  }
}

We keep the function small. Robots.txt is a poor place for complex business logic or remote dependencies.

We understand cache and dynamic behavior

Generated metadata routes can be cached depending on how they are implemented. If output depends on request-time state or deployment configuration, we verify the current Next.js behavior instead of assuming a code edit immediately changes the public response.

We protect previews outside robots.txt

Vercel preview or staging hosts should not rely only on Disallow: /. Private previews need authentication or access controls. For public previews that must not appear in search, we also review noindex behavior.

We keep sitemap hostnames production-safe

A generated route can accidentally emit a preview domain if the base URL comes from the wrong environment variable. We inspect the final Sitemap: URL and canonical production hostname after every deployment.

We verify the built response

We do not stop at TypeScript correctness. We open /robots.txt on the deployed host, inspect the response, run the validator, and test representative URLs with the tester. If output differs from the code, we diagnose routing, cache, environment variables, and deployment ownership.

Our Next.js checklist

1. choose app/robots.txt for a genuinely static policy;
2. use app/robots.ts or app/robots.js only when generation adds value;
3. type generated output with MetadataRoute.Robots;
4. keep sitemap URLs absolute and production-canonical;
5. understand cache/dynamic behavior;
6. protect preview environments with real access control;
7. verify the public file after deployment.

The best Next.js robots implementation is usually the least dynamic version that still matches the deployment architecture.