---
title: "Write a crawlable robots.txt"
description: "robots.txt controls crawling, not indexing. Serve one file per host at the root with a 200 status and a sitemap reference, allow CSS and JavaScript so rendering matches users, and block only infinite or private URL spaces. A single Disallow: / line can remove the entire site from search."
url: "https://seo-mcp.io/t/write-a-crawlable-robots-txt"
---

# Write a crawlable robots.txt

robots.txt controls crawling, not indexing. Serve one file per host at the root with a 200 status and a sitemap reference, allow CSS and JavaScript so rendering matches users, and block only infinite or private URL spaces. A single Disallow: / line can remove the entire site from search.

## Prerequisites

- Control over the domain root

## Steps

1. **Serve one valid file at the root** — Return robots.txt from https://your-host/robots.txt with status 200 and content-type text/plain. Do not redirect it and do not serve a cached HTML error page.
   Check: curl -sI your production domain/robots.txt returns 200 and content-type text/plain.
2. **Allow rendering resources and reference the sitemap** — Do not disallow CSS, JavaScript or image files needed to render pages. Add a Sitemap line with an absolute HTTPS URL near the top of the file.
   Check: robots.txt contains a Sitemap: line whose URL returns 200.
3. **Block only infinite or private spaces** *(general best practice)* — Disallow crawl paths that generate unlimited URLs, such as internal search or sort parameters. Never ship Disallow: / outside a staging environment.
   Check: grep -c '^Disallow: /$' robots.txt returns 0 in production.

## Source

@seo-mcp on 2026-10-19: https://developers.google.com/search/docs/crawling-indexing/robots/intro

Canonical: https://seo-mcp.io/t/write-a-crawlable-robots-txt