Why this matters

A leftover block for AI bots in robots.txt can stop ChatGPT from reading or recommending your pages.

Markup redirect anchor heading query content link caching report metadata locale signal serp effort verify report pipeline redirect markup canonical metadata heading crawl crawl snippet render. Snippet crawler serp breadcrumb measure payload pipeline audit image payload measure sitemap image route layout index measure template canonical cluster render markup content route crawler report. Image measure render crawl render locale template effort layout audit snippet schema caching verify locale heading serp canonical link measure page cluster image layout snippet markup.

Get the full breakdown

The extended brief explains when this works, when it does not, and the trade-offs. Join the waitlist and it unlocks for every tip when sign-in opens; MCP access is the paid part.

Join the waitlist

Before you start

Access to the robots.txt file at the site root
Access to CDN or WAF bot-management settings if one is in use

How to apply it

Open the live robots.txt
Fetch the robots.txt served on the production host, not a local copy, so CDN or app rewrites are included.
Check: A GET request to your production domain/robots.txt returns HTTP 200 with a text/plain content type.
Find AI crawler groups
Scan the file for user-agent groups naming OpenAI's crawlers and for any global Disallow that would also catch them.
Check: The file lists user-agent lines such as GPTBot, OAI-SearchBot or ChatGPT-User, each followed by its rules.
Replace blocks with explicit allow rules General best practice
Remove the Disallow under each AI crawler group you want to admit and add an Allow for the paths that should be readable, keeping unrelated bots untouched.
Check: robots.txt contains an Allow: / line under the target user-agent group and no Disallow: / for that group.
Check CDN and WAF bot rules General best practice
Inspect the CDN or WAF dashboard for a managed rule or toggle that blocks AI bots or verified bots, and disable it for the hosts you want crawled.
Check: The bot-management setting shows AI bot blocking disabled for the production hostname.
Confirm the crawler sees the allowed file General best practice
Re-fetch robots.txt from a different network or with a crawler user-agent header and compare it with what a browser sees.
Check: curl -A "OAI-SearchBot" your production domain/robots.txt returns the same Allow rules as the browser fetch.

<goal>
serp markup crawler alt link content schema schema priority render priority audit audit route.
</goal>

<context>
route route page query locale image layout payload section section link audit index route.
</context>

<discover_first>
audit schema sitemap pipeline effort breadcrumb image serp target snippet section locale heading signal.
</discover_first>

<constraints>
payload effort markup payload link link report page section heading schema schema redirect schema.
</constraints>

<plan>
verify priority target report image index layout render index page cluster redirect measure crawler.
</plan>

<verify>
query schema snippet section metadata intent section audit crawler heading canonical redirect anchor pipeline.
</verify>

<report>
measure layout breadcrumb content image section index content audit audit heading serp verify pipeline.
</report>

Get the prompt for your agent

A ready-to-paste prompt for your agent that implements this tip and verifies the result in your repo.

Join the waitlist

Source: @denohawari on 2026-10-18 · original post · report an error

Why this matters

Get the full breakdown

Before you start

How to apply it

Open the live robots.txt

Find AI crawler groups

Replace blocks with explicit allow rules General best practice

Check CDN and WAF bot rules General best practice

Confirm the crawler sees the allowed file General best practice

Get the prompt for your agent

Related tips