---
title: "#robots — seo-mcp"
description: "SEO tips tagged Robots."
url: "https://seo-mcp.io/tag/robots"
---

# #robots

5 published tips tagged “Robots”.

- [Allow AI crawlers in robots](https://seo-mcp.io/t/allow-ai-crawlers-in-robots) — AI assistants fetch pages with their own user agents: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, and Google-Extended for Gemini. Decide per bot between training and retrieval, write explicit groups, and test with curl using each agent string.
- [Ship app/sitemap.ts and app/robots.ts](https://seo-mcp.io/t/ship-app-sitemap-ts-and-app-robots-ts) — Next.js App Router turns app/sitemap.ts and app/robots.ts into /sitemap.xml and /robots.txt at build time. Return only canonical, indexable URLs from sitemap.ts with honest lastModified values, and set the rules plus the sitemap location from robots.ts. Both files must be reachable without authentication and must not be shadowed by static files in public/.
- [Write a crawlable robots.txt](https://seo-mcp.io/t/write-a-crawlable-robots-txt) — robots.txt controls crawling, not indexing. Serve one file per host at the root with a 200 status and a sitemap reference, allow CSS and JavaScript so rendering matches users, and block only infinite or private URL spaces. A single Disallow: / line can remove the entire site from search.
- [Allow AI crawlers in robots.txt so ChatGPT can cite the site](https://seo-mcp.io/t/allow-ai-crawlers-in-robots-txt-so-chatgpt-can-cite-the-site) — Review robots.txt for Disallow rules aimed at AI crawler user agents, which many sites added in 2024, and replace them with explicit Allow rules for the pages you want cited. Then check that your CDN or WAF does not block the same bots at the edge, since a robots.txt fix alone will not help if the request never reaches the origin.
- [Audit Googlebot crawl behavior with server logs to find crawl traps](https://seo-mcp.io/t/audit-googlebot-crawl-behavior-with-server-logs-to-find) — Server log analysis reveals how Googlebot actually spends crawl budget, exposing infinite URL spaces, thin faceted combinations and orphan pages that internal tools miss. The tip walks through filtering verified Googlebot requests, grouping them by URL pattern, then using robots rules, canonicals and internal links to redirect crawl attention toward pages that earn traffic.