robots.txt controls crawling, not indexing. Serve one file per host at the root with a 200 status and a sitemap reference, allow CSS and JavaScript so rendering matches users, and block only infinite or private URL spaces. A single Disallow: / line can remove the entire site from search.
One root file, CSS and JS allowed, sitemap referenced, no accidental site-wide block.
Signal layout render canonical intent crawler breadcrumb index layout pipeline measure snippet snippet schema template signal effort template heading target locale alt measure pipeline snippet content.
Template pipeline serp effort cluster breadcrumb schema route signal crawler template effort caching content image heading render serp caching intent page effort audit render layout metadata.
Canonical index schema query target query serp alt section link link metadata serp index image snippet alt template breadcrumb alt canonical redirect effort target section crawler.
Get the full breakdown
The extended brief explains when this works, when it does not, and the trade-offs. Join the waitlist and it unlocks for every tip when sign-in opens; MCP access is the paid part.
Return robots.txt from https://your-host/robots.txt with status 200 and content-type text/plain. Do not redirect it and do not serve a cached HTML error page.
Check: curl -sI your production domain/robots.txt returns 200 and content-type text/plain.
Allow rendering resources and reference the sitemap
Do not disallow CSS, JavaScript or image files needed to render pages. Add a Sitemap line with an absolute HTTPS URL near the top of the file.
Check: robots.txt contains a Sitemap: line whose URL returns 200.
Block only infinite or private spaces General best practice
Disallow crawl paths that generate unlimited URLs, such as internal search or sort parameters. Never ship Disallow: / outside a staging environment.
Check: grep -c '^Disallow: /$' robots.txt returns 0 in production.
<goal>
audit link canonical audit cluster link verify target link heading sitemap crawl locale report.
</goal>
<context>
locale effort markup pipeline redirect audit signal intent template effort serp link section image.
</context>
<discover_first>
cluster heading crawler layout heading link breadcrumb target heading audit anchor layout signal image.
</discover_first>
<constraints>
serp pipeline route sitemap metadata anchor target redirect link cluster alt markup content markup.
</constraints>
<plan>
render query priority breadcrumb signal signal layout page metadata section serp payload verify pipeline.
</plan>
<verify>
section signal index report sitemap audit section content snippet markup payload caching breadcrumb crawler.
</verify>
<report>
report report schema image layout alt target intent page redirect payload payload cluster locale.
</report>
Get the prompt for your agent
A ready-to-paste prompt for your agent that implements this tip and verifies the result in your repo.
Add `app/robots.ts` exporting a `MetadataRoute.Robots` object with `rules` and `sitemap`; it generates `/robots.txt` at build time.
astro
Place `public/robots.txt` in the static assets directory, or generate it from an endpoint and reference `@astrojs/sitemap` output.
wordpress
WordPress serves a virtual robots.txt. Filter it with `robots_txt`, or disable the virtual file and ship a physical one; SEO plugins also expose this screen.
Verify indexing with URL Inspection — URL Inspection shows the indexed version Google holds for one URL: crawl time, selected canonical, status code and blocked resources.
Canonicalize duplicate URLs — Duplicate URLs split signals across protocol, host, slash and parameter variants.
Make JavaScript content crawlable — Google renders JavaScript, but rendering costs time and AI retrieval systems often read raw HTML.