Ship app/sitemap.ts and app/robots.ts

Why this matters

Two file conventions give crawlers the map and the rules.

Before you start

• Next.js App Router project with a published-content data source
• A single production site URL constant

How to apply it

1. Create app/sitemap.ts

   Export a default function returning MetadataRoute.Sitemap. Map published records to { url, lastModified }, using absolute URLs from siteUrl and the record updated timestamp rather than the build time.

   Check: curl -s your production domain/sitemap.xml returns 200, is valid XML, and lists only canonical published URLs.

2. Split large catalogs with generateSitemaps

   When the set exceeds 50,000 URLs or 50 MB, export generateSitemaps() returning ids and emit one file per id. Next.js then serves /sitemap/<id>.xml and an index at /sitemap.xml.

   Check: curl -s your production domain/sitemap.xml returns a sitemap index and every child loc returns 200.

3. Add app/robots.ts

   Return rules with userAgent '*', allow '/' and disallow only paths with no search value. Set sitemap to siteUrl + '/sitemap.xml' and host to the production origin so one file documents the whole crawl policy.

   Check: curl -s your production domain/robots.txt returns 200, contains a Sitemap line and disallows no indexable path.

4. Remove shadowing static files

   Delete public/robots.txt and public/sitemap.xml plus any proxy or edge rewrite for those paths. A static file or rewrite can serve instead of the metadata route and silently drop your changes.

   Check: curl -sI your production domain/robots.txt returns 200 with text/plain, and the repo has no public/robots.txt.

5. Submit and monitor

   Submit the sitemap in Search Console and compare discovered URLs with indexed ones. Confirm robots.txt does not block CSS, JavaScript or any path that must rank.

   Check: The Search Console Sitemaps report lists the file as Success with a discovered URL count above zero.

<goal>
heading breadcrumb pipeline verify locale canonical template priority index route payload markup redirect heading.
</goal>

<context>
breadcrumb crawl crawler heading intent template anchor section crawler markup markup section payload priority.
</context>

<discover_first>
crawler breadcrumb index schema layout link page audit locale image link measure sitemap section.
</discover_first>

<constraints>
target section sitemap cluster image page target layout schema query intent sitemap metadata report.
</constraints>

<plan>
crawler index query metadata markup verify canonical breadcrumb audit render sitemap index caching intent.
</plan>

<verify>
snippet route locale render breadcrumb report redirect alt index markup effort serp heading signal.
</verify>

<report>
image report target section audit render audit page query signal caching heading audit crawl.
</report>

Source: seo-mcp playbook · curated entry · report an error