A robots.txt written in 2023 probably doesn't name the crawlers AI companies run today, and several companies now run separate agents for training, search and user requests. This guide gives you the directives for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest of the 2026 lineup, and shows how to keep search traffic while opting out of training.
How do I block AI crawlers with robots.txt?
Add a User-agent block in your robots.txt file for each AI crawler you want to block, followed by Disallow: /. The 2026 essentials are GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Bytespider. Place the file at your domain root and verify with each vendor's published agent documentation.
Why robots.txt still matters in 2026
The Robots Exclusion Protocol dates from 1994 and became an IETF standard, RFC 9309, in September 2022. OpenAI, Anthropic, Google, Apple, Perplexity, Amazon and Common Crawl all publish their crawler tokens, and their documentation says their automated crawlers follow robots.txt.
robots.txt is only a request, though. Crawlers that follow the rules obey it and others ignore it. Real enforcement takes WAF rules or IP-level blocks, but robots.txt is the first step and the opt-out the major vendors document.
What changed between 2023 and 2026
- Separate agents for training, search and user requests. OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User) and Anthropic (ClaudeBot, Claude-SearchBot, Claude-User) split these jobs, and Perplexity separates PerplexityBot from Perplexity-User. Blocking one doesn't block the others.
- Training opt-outs that leave search alone. Google-Extended and Applebot-Extended are robots.txt tokens, not separate crawlers, so you can stay in search while opting out of AI training. Bing has no such token and uses NOCACHE and NOARCHIVE meta tags instead.
- More published agents. Meta (meta-externalagent), Amazon (Amazonbot) and DuckDuckGo (DuckAssistBot) all document theirs.
The 2026 AI crawler block list
Drop the following directives into your robots.txt file. Each block uses a separate User-agent declaration. Listing several agents on one line, separated by commas, is not part of the spec; RFC 9309 does allow several User-agent lines in a row above one set of rules.
# OpenAI — model training
User-agent: GPTBot
Disallow: /
# OpenAI — real-time browsing for ChatGPT users
User-agent: ChatGPT-User
Disallow: /
# OpenAI: search indexing (ChatGPT search)
User-agent: OAI-SearchBot
Disallow: /
# Anthropic: model training
User-agent: ClaudeBot
Disallow: /
# Anthropic: search indexing for Claude
User-agent: Claude-SearchBot
Disallow: /
# Anthropic: pages fetched when a Claude user asks
User-agent: Claude-User
Disallow: /
# Perplexity
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
# Google — AI training opt-out (does not affect Search)
User-agent: Google-Extended
Disallow: /
# Apple — Apple Intelligence training opt-out
User-agent: Applebot-Extended
Disallow: /
# Common Crawl — feeds most open-source training sets
User-agent: CCBot
Disallow: /
# ByteDance / TikTok
User-agent: Bytespider
Disallow: /
# Meta: AI training and indexing
User-agent: Meta-ExternalAgent
Disallow: /
# Amazon: Amazon says this data may train its AI models
User-agent: Amazonbot
Disallow: /
# Cohere
User-agent: cohere-ai
Disallow: /
# DuckDuckGo AI assistant
User-agent: DuckAssistBot
Disallow: /
Not every agent on that list trains models. OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages for AI search features, and Perplexity and DuckDuckGo both say their bots' data isn't used for model training, so delete those groups if you want to appear in AI search. Our robots.txt generator builds the base file (the User-agent: * group, your paths and the Sitemap line) with a one-click GPTBot block; paste the other groups underneath before you upload. Google ignores invalid lines, including a byte order mark, so a typo usually costs you one rule rather than the file.
Which AI crawlers to block and which to allow
The vendors' own documentation splits the agents into three jobs:
| Job | User agents | What blocking it costs you |
|---|---|---|
| Model training | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent | Nothing in search, according to OpenAI, Google and Apple; it stops future collection for training |
| AI search indexing | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Your pages stop appearing as sources in those AI search tools |
| User-requested fetches | ChatGPT-User, Claude-User, Perplexity-User | A user's request for your page fails, where the fetcher honors robots.txt at all |
If you want AI search traffic but not training use, disallow the first row and leave the second alone. OpenAI says opted-out sites "will not be shown in ChatGPT search answers," and Perplexity recommends allowing PerplexityBot "to ensure your site appears in search results." The ChatGPT search checklist and the Perplexity SEO guide cover those two bots in detail.
Block training, allow search: the surgical approach
Most publishers don't want to disappear from Google or Bing; they want to stop their content being used to train AI models. That distinction is now possible:
# Keep Google Search indexing
User-agent: Googlebot
Allow: /
# But opt out of Gemini training and grounding
User-agent: Google-Extended
Disallow: /
# Keep Bing Search
User-agent: Bingbot
Allow: /
# Apple Search stays, Apple Intelligence training blocked
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Disallow: /
Google says Google-Extended does not affect inclusion or ranking in Search, and Apple says pages that disallow Applebot-Extended still appear in its search results. AI Overviews and AI Mode are part of Search, so Google-Extended isn't the control for them; Google points to nosnippet, max-snippet and noindex instead. For Bing, the NOCACHE or NOARCHIVE meta tag limits use in its AI answers while the page stays in search.
Per-path blocks: protecting premium content only
Site-wide blocks can be too blunt. If you run a marketing site with a paywalled archive, you may want to expose the front end but seal off the archive:
User-agent: GPTBot
Disallow: /archive/
Disallow: /premium/
Disallow: /members/
User-agent: ClaudeBot
Disallow: /archive/
Disallow: /premium/
Disallow: /members/Rule order inside a group doesn't matter. RFC 9309 says crawlers use the most specific match, meaning the rule with the most characters, and Google uses the least restrictive rule when an Allow and a Disallow match equally. For people reading the file, group rules by agent, comment each group, and avoid trailing whitespace.
Common mistakes that silently fail
A named group overrides the wildcard group
A crawler that finds a group with its own name follows only that group and ignores User-agent: *; RFC 9309 makes * the fallback, and Google says the two are not combined. If your * group disallows /private/ and your GPTBot group only disallows /archive/, GPTBot may crawl /private/, so repeat shared rules in every named group. And User-agent: * with Disallow: / does block GPTBot when it has no group of its own, along with every search engine.
Meta robots tags don't help here
The noindex meta tag and X-Robots-Tag header control search indexing. OpenAI's and Google's training opt-outs are robots.txt tokens, and neither documents noindex as a training control, so use robots.txt for ingestion and meta robots for indexing. The exceptions are Bing (NOCACHE, NOARCHIVE) and Amazon, which treats noarchive as "do not use the page for model training".
Caching and CDN propagation
Crawlers cache the file: Google for up to 24 hours, Amazonbot up to 30 days, and DuckDuckGo says a DuckAssistBot opt-out takes 72 hours. If your site sits behind Cloudflare, Fastly, or a similar CDN, purge the cache for /robots.txt immediately after editing, and verify with curl -A "GPTBot" https://yourdomain.com/robots.txt from a clean network.
Verifying your block list works
Three checks before you call it done:
- Fetch the file directly.
curl https://yourdomain.com/robots.txtshould return the file with HTTP 200, content typetext/plain, and no HTML wrapping. - Check the vendors' own references. OpenAI, Anthropic, Perplexity, Amazon, DuckDuckGo and Common Crawl publish their crawlers' IP ranges, so you can confirm a request claiming to be GPTBot really is. For Google, the robots.txt report in Search Console (which replaced the robots.txt Tester) shows the version Google fetched and any errors.
- Monitor your logs. Filter access logs for the agent strings above. A correctly blocked bot will still hit
/robots.txt(that's how it learns it's blocked) but should not request other URLs afterward.
To see what a crawler gets when it asks for the file, run your robots.txt URL through the redirect checker. It shows each hop and its status code, so you'll spot a /robots.txt that redirects to the home page or to another host. Google follows at least five redirect hops and then treats the file as a 404, which means no rules at all, so serve it directly with a 200. While you're there, check that the Sitemap line points to a real file; the XML sitemap generator can build one if you don't have it.
When robots.txt isn't enough
For sensitive content (paid archives, member-only material, proprietary data), add other defenses:
- WAF rules blocking the published IP ranges of major AI vendors (OpenAI, Anthropic, and Perplexity all publish these, though Anthropic warns that IP blocking may not work correctly and recommends robots.txt).
- Cloudflare's AI crawler controls, which let you block training, search and agent crawlers separately from the dashboard.
- Authentication. Anything truly valuable should require a login, not a politeness directive.
FAQ
Does blocking GPTBot remove my content from ChatGPT?
No. It stops future crawling for training and doesn't remove anything already collected. ChatGPT search uses OAI-SearchBot, and OpenAI says robots.txt rules may not apply to ChatGPT-User because a person initiated the visit.
Will blocking Google-Extended hurt my SEO?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It governs whether your content may train future Gemini models and ground answers in Gemini Apps and Vertex AI; AI Overviews follow Search's snippet controls instead.
How often should I update my robots.txt for AI crawlers?
Check it a few times a year. Anthropic no longer lists the Claude-Web and anthropic-ai tokens older guides include, OpenAI has added OAI-AdsBot, and Google plans to support a shared AI-training signal (user-agent: __usage_ai-training) in early 2027.
Do AI crawlers actually respect robots.txt?
The vendors say their automated crawlers do. User-requested fetchers are the exception: ChatGPT-User, Perplexity-User and Meta's meta-externalfetcher may skip it, according to their own docs. Scrapers that hide their identity ignore it entirely, so combine robots.txt with WAF rules and authentication for high-value content.
Does blocking AI crawlers hurt my SEO?
Blocking training-only tokens such as GPTBot, Google-Extended or Applebot-Extended doesn't affect Google Search, according to Google and Apple. Blocking search agents is different: disallow OAI-SearchBot or PerplexityBot and you drop out of ChatGPT or Perplexity search answers. The changes worth watching each year are in AI SEO trends for 2026.
Can I block AI crawlers from specific countries or regions?
Robots.txt does not support geo-targeting; it works per user agent only. For regional blocks, use your CDN's geo-fencing or WAF rules.
What's the difference between ClaudeBot, Claude-SearchBot and Claude-User?
ClaudeBot collects content that may be used to train Anthropic's models, Claude-SearchBot crawls to improve Claude's search results, and Claude-User fetches pages when a user asks. Blocking one does not block the others.
