---
title: "Allow AI crawlers in robots"
description: "AI assistants fetch pages with their own user agents: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, and Google-Extended for Gemini. Decide per bot between training and retrieval, write explicit groups, and test with curl using each agent string."
url: "https://seo-mcp.io/t/allow-ai-crawlers-in-robots"
---

# Allow AI crawlers in robots

AI assistants fetch pages with their own user agents: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, and Google-Extended for Gemini. Decide per bot between training and retrieval, write explicit groups, and test with curl using each agent string.

## Prerequisites

- Control of robots.txt
- A decision on training versus retrieval

## Steps

1. **Check the inherited policy** — Read the User-agent: * group. If it contains Disallow: /, every AI crawler is blocked regardless of named groups you add later.
   Check: The star group contains no site-wide disallow rule.
2. **Write explicit groups per bot** *(general best practice)* — Add separate groups for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended with Allow rules for content and rendering assets.
   Check: robots.txt contains a named group for each bot with no site-wide disallow.
3. **Test each agent string** *(general best practice)* — Request a key page with curl using each user agent and confirm a 200 response with full HTML.
   Check: curl -A "OAI-SearchBot" your production domain/ returns 200 and the page HTML.

## Source

@seo-mcp on 2026-10-19: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

Canonical: https://seo-mcp.io/t/allow-ai-crawlers-in-robots