---
title: "Allow AI crawlers in robots.txt so ChatGPT can cite the site"
description: "Review robots.txt for Disallow rules aimed at AI crawler user agents, which many sites added in 2024, and replace them with explicit Allow rules for the pages you want cited. Then check that your CDN or WAF does not block the same bots at the edge, since a robots.txt fix alone will not help if the request never reaches the origin."
url: "https://seo-mcp.io/t/allow-ai-crawlers-in-robots-txt-so-chatgpt-can-cite-the-site"
---

# Allow AI crawlers in robots.txt so ChatGPT can cite the site

Review robots.txt for Disallow rules aimed at AI crawler user agents, which many sites added in 2024, and replace them with explicit Allow rules for the pages you want cited. Then check that your CDN or WAF does not block the same bots at the edge, since a robots.txt fix alone will not help if the request never reaches the origin.

## Prerequisites

- Access to the robots.txt file at the site root
- Access to CDN or WAF bot-management settings if one is in use

## Steps

1. **Open the live robots.txt** — Fetch the robots.txt served on the production host, not a local copy, so CDN or app rewrites are included.
   Check: A GET request to your production domain/robots.txt returns HTTP 200 with a text/plain content type.
2. **Find AI crawler groups** — Scan the file for user-agent groups naming OpenAI's crawlers and for any global Disallow that would also catch them.
   Check: The file lists user-agent lines such as GPTBot, OAI-SearchBot or ChatGPT-User, each followed by its rules.
3. **Replace blocks with explicit allow rules** *(general best practice)* — Remove the Disallow under each AI crawler group you want to admit and add an Allow for the paths that should be readable, keeping unrelated bots untouched.
   Check: robots.txt contains an Allow: / line under the target user-agent group and no Disallow: / for that group.
4. **Check CDN and WAF bot rules** *(general best practice)* — Inspect the CDN or WAF dashboard for a managed rule or toggle that blocks AI bots or verified bots, and disable it for the hosts you want crawled.
   Check: The bot-management setting shows AI bot blocking disabled for the production hostname.
5. **Confirm the crawler sees the allowed file** *(general best practice)* — Re-fetch robots.txt from a different network or with a crawler user-agent header and compare it with what a browser sees.
   Check: curl -A "OAI-SearchBot" your production domain/robots.txt returns the same Allow rules as the browser fetch.

## Source

@denohawari on 2026-10-18: https://x.com/denohawari/status/2111174588847567186

Canonical: https://seo-mcp.io/t/allow-ai-crawlers-in-robots-txt-so-chatgpt-can-cite-the-site