Should You Block AI Crawlers? A robots.txt Guide to GPTBot, ClaudeBot and PerplexityBot

There are two decisions here and almost everyone makes them as one. The first is whether your content may be used to train models. The second is whether your pages may be retrieved and cited when someone asks an assistant a question. Every major vendor now ships a separate user agent for each, so you can say yes to one and no to the other.

The practical answer for most sites: block the training crawlers if you object to training, allow the search crawlers, and check that your CDN is not quietly overruling both. The expensive mistake is a blanket disallow that takes you out of AI answers as a side effect of an opinion about training data.

Which crawler does what?

Straight from each vendor's own documentation.

The pattern is consistent across all four vendors: the training crawler and the search crawler are different agents, and only the second one is connected to being cited.

Two agents in that table break the model, and both are the user-triggered fetchers. OpenAI says robots.txt rules "may not apply" to ChatGPT-User because a person initiated the request; Perplexity says the same of Perplexity-User. Whether you consider that reasonable depends on whether you think a person pasting your URL into an assistant is a reader or a robot. Either way, robots.txt is not the lever for those two.

What does Google-Extended actually control?

Less than its name suggests, and this is the single most misunderstood entry in the table. Google's crawler documentation says Google-Extended governs "whether content Google crawls from their sites may be used for training future generations of Gemini models", and then, in a sentence worth reading twice:

Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.

It is also not a crawler. Googlebot does the fetching; Google-Extended is a token that governs downstream use. Which means AI Overviews and AI Mode are unaffected by it, because they draw on the ordinary Search index. If you want out of AI Overviews, Google-Extended is not the control, and the AI Overviews guide covers what is.

Is robots.txt enough?

It is a convention, not a fence. The standard is RFC 9309, and compliance is voluntary. Well-behaved crawlers honour it; nothing forces anyone to.

How well it holds became a public argument in August 2025, when Cloudflare accused Perplexity of using undeclared crawlers with generic browser user agents to fetch pages from sites that had blocked PerplexityBot, and de-listed it as a verified bot. Perplexity called the report a publicity stunt and said Cloudflare had conflated user-initiated fetches with crawling. It has not been independently adjudicated, and it is worth knowing about mostly as a reminder that robots.txt is a request.

If you need enforcement rather than a request, that is a job for your CDN or web server: bot management rules, rate limits or an outright block by user agent and IP. The corollary matters too. Your CDN may already be blocking AI crawlers without you asking. Cloudflare began blocking AI crawlers by default for new domains on 1 July 2025 and launched a Pay Per Crawl beta alongside it, so "I never blocked anything" is not the same as "nothing is blocked". Check the edge, not just the file.

Three stances you can copy

Allow everything. No rules needed. You are in every training set and every answer engine.

Block training, allow search. The common choice for a site that wants citations but objects to being training data:

User-agent: GPTBot
Disallow: /
 
User-agent: ClaudeBot
Disallow: /
 
User-agent: CCBot
Disallow: /
 
User-agent: Google-Extended
Disallow: /
 
User-agent: OAI-SearchBot
Allow: /
 
User-agent: Claude-SearchBot
Allow: /
 
User-agent: PerplexityBot
Allow: /

Block everything AI. Add Disallow: / blocks for OAI-SearchBot, Claude-SearchBot and PerplexityBot as well. Understand what you are buying: you disappear from AI answers, and the referral traffic you give up is real but small. Across the published crawl-and-refer measurements the direction is consistent, if the exact figures move month to month: Google sends far more visits per page crawled than any of the AI vendors do. Blocking a search engine costs you traffic. Blocking an AI answer engine costs you attribution more than traffic.

Whichever you pick, verify it afterwards by watching your logs for the agent names in the table above, not by re-reading your own file.

What is changing?

Two efforts are trying to replace the current mess of vendor-specific tokens with something a publisher can state once.

The IETF's AI preferences working group is standardising a vocabulary for exactly this split, with separate categories for training and for search, expressible in robots.txt and in an HTTP header. The vocabulary draft has been adopted by the working group and is on the standards track; it is not a finished standard, and nothing implements it yet.

Really Simple Licensing, launched on 10 September 2025 with a 1.0 specification in December 2025, goes further and lets a publisher attach licensing and compensation terms in robots.txt. It has substantial publisher backing. Whether AI companies pay is a separate question from whether the file parses.

Neither changes what you should do this week. Both are worth watching if you publish at scale.

Using the Revised directory for this

One place this matters that nobody mentions: a domain you acquire arrives with somebody else's crawler policy attached.

If you register an expired domain and point it at new hosting, the old robots.txt is gone with the old host, which is usually what you want. But two things travel further than the file. An X-Robots-Tag header configured at a CDN can survive a nameserver change if you inherit any of the old setup, and a domain that spent two years parked has been serving a parking page to every crawler that visited, including the search agents in the table above. Either way the first day's work on a newly registered domain is the same: publish a robots.txt that allows the search crawlers, publish something real, and check the logs a week later to confirm the agents are arriving.

When you are choosing which domain to register in the first place, our directory publishes the evidence on every listing before you reveal a name: the referring-domain band, the sources linking in by name, the archived history, the spam and trademark screen, and when availability was last checked. Filter under Quality on a minimum Agent Citability score to sort by how deeply a domain is referenced by the kinds of sources answer engines cite, and under History to require a real archived record rather than an old registration date. After you reveal one, the expired domain checker is the quickest way to see its current DNS and registry state before you point anything anywhere.

FAQ

Should I block AI crawlers? Block the training crawlers if you object to training on your content. Do not block the search crawlers unless you have decided you want no presence in AI answers, because that is exactly what it buys.

Does blocking GPTBot stop me appearing in ChatGPT? No. GPTBot is the training crawler. OAI-SearchBot is the one connected to ChatGPT's search answers, and OpenAI says sites opted out of it "will not be shown in ChatGPT search answers".

Does Google-Extended keep me out of AI Overviews? No. Google's documentation says it does not affect inclusion in Google Search or ranking, and AI Overviews draw on the Search index.

Do AI crawlers respect robots.txt? The declared ones generally do. The user-triggered fetchers explicitly may not, by their vendors' own documentation, and there has been at least one high-profile contested allegation of a vendor evading blocks. Enforcement requires your CDN, not your robots.txt.

Will blocking AI crawlers hurt my SEO? Not in Google Search, provided you keep Googlebot allowed and do not block Google-Extended thinking it is a search control. The cost is presence in AI answers, which is a separate thing from rankings.

Is my CDN blocking AI crawlers already? Possibly. Cloudflare turned on default AI-crawler blocking for new domains in July 2025. Check your bot management settings before concluding your robots.txt is the whole policy.