Blocking AI crawlers means adding
Disallowrules for each bot’s user-agent to robots.txt, and, if you want real enforcement, blocking the same user-agents at the firewall or CDN level too. There are three classes of AI bot to decide on separately: training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended), search/answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and on-demand bots that fire when a user asks a live question inside ChatGPT, Claude, or Perplexity (ChatGPT-User, Claude-User, Perplexity-User). robots.txt is a request, not a lock: OpenAI’s own documentation says ChatGPT-User does not respect it, and Perplexity’s docs say the same for Perplexity-User, so a robots.txt line alone will not stop every fetch. This site currently blocks none of them, and explains why below.
Every AI company that crawls the web publishes a user-agent string, and every one of those strings can be blocked with a line in robots.txt. What most guides skip is that “block AI crawlers” isn’t one decision – it’s several, because training bots, search bots, and on-demand bots behave differently, and blocking one doesn’t touch the others.
The three classes of AI crawler
Training bots crawl to build or update a model’s training data. GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended fall here. Blocking these stops your content from being used in future model training, and nothing else – it doesn’t touch whether your pages appear in AI search results or chat answers today.
Search and answer bots crawl to index pages for retrieval, the same way a search engine does, so a model can cite or summarize your page in response to a live question. OAI-SearchBot, Claude-SearchBot, and PerplexityBot are in this class. Blocking these is closer to blocking Googlebot: it removes you from that assistant’s retrieval index entirely.
On-demand bots fire in real time when a specific user action triggers a fetch – someone pastes your URL into ChatGPT, or asks Perplexity a question that leads it to open your page live. ChatGPT-User, Claude-User, and Perplexity-User do this. The important nuance: OpenAI’s documentation states ChatGPT-User does not respect robots.txt because the action is user-initiated, and Perplexity’s documentation says its user-triggered fetcher “generally ignores robots.txt rules” for the same reason. Claude-User is the exception – Anthropic’s documentation confirms it does respect robots.txt.
Every current AI crawler user-agent
This list reflects vendor documentation as of this writing; user-agents change, so recheck before you rely on it.
- GPTBot (OpenAI) – training data crawler, respects robots.txt.
- OAI-SearchBot (OpenAI) – crawls to surface results inside ChatGPT search, respects robots.txt.
- OAI-AdsBot (OpenAI) – crawls for ad-quality checks, respects robots.txt.
- ChatGPT-User (OpenAI) – fires on live user actions inside ChatGPT, does not respect robots.txt.
- ClaudeBot (Anthropic) – training data crawler, respects robots.txt.
- Claude-SearchBot (Anthropic) – crawls for search-result indexing, respects robots.txt.
- Claude-User (Anthropic) – fetches pages in response to a live Claude query, respects robots.txt.
- PerplexityBot (Perplexity) – search indexing crawler, respects robots.txt.
- Perplexity-User (Perplexity) – fetches pages for a live user query, does not respect robots.txt.
- Google-Extended (Google) – governs use of your content for training Gemini apps and the Vertex AI API for Gemini; does not affect Google Search ranking or inclusion.
- Applebot-Extended (Apple) – a secondary signal that controls AI-training use of pages Applebot already crawled; disallowing it does not block Applebot itself or remove you from Siri or Spotlight results.
- CCBot (Common Crawl) – the crawler behind the Common Crawl dataset many models train on, checks robots.txt before crawling.
To block every bot above in robots.txt, add a stanza per user-agent:
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: OAI-AdsBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /robots.txt vs. firewall-level blocking
robots.txt is a published request that well-behaved bots choose to follow. That’s true for most of the bots above, by their own vendors’ documentation. But two of the three on-demand bots – ChatGPT-User and Perplexity-User – explicitly don’t honor it, because the vendors treat a live user click as different from an automated crawl. If you actually need to stop those fetches, a robots.txt line does nothing; you need to block the user-agent (or the originating IP ranges, where published) at the CDN or firewall layer, so the request is refused before it reaches your server. robots.txt is worth setting regardless, since most bots do respect it, but treat it as a courtesy notice for the well-behaved majority, not an enforcement mechanism for all of them.
What each block actually costs you
Blocking training bots (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) costs you nothing visible today. Your pages keep ranking in Google, keep showing up in AI search answers, and keep getting cited by assistants that use separate crawlers for retrieval. What you give up is any influence your content might have had on a future model’s training data – a cost with no way to measure its size, since no vendor publishes how much any one site’s content matters to training outcomes.
Blocking search and answer bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) has a direct, visible cost: your pages stop being eligible for citation or retrieval in that assistant’s live answers. If a goal for your site is showing up when someone asks ChatGPT or Perplexity a question your content answers, blocking these bots removes you from consideration entirely, the same way blocking Googlebot removes you from Google Search.
Blocking on-demand bots (ChatGPT-User, Claude-User, Perplexity-User) stops individual users from getting your page fetched live when they paste your URL or ask a question that leads there – a smaller, harder-to-measure slice of traffic, and one that’s already partly unenforceable through robots.txt alone for two of the three vendors.
Why this site doesn’t block AI crawlers
As of this writing, this site’s own robots.txt has no AI-bot rules at all – you can check it directly at organicos.shivaatripathi.com/robots.txt. That’s a deliberate choice, not an oversight. The angle this site runs on is visibility in AI-driven search and chat answers, covered in more detail in how to get cited by ChatGPT and Perplexity; blocking search and answer bots would directly undercut that goal, and blocking training bots would trade an unmeasurable future benefit for a real, current cost in reduced reach. If your site depends on subscription content, proprietary data, or a business model where AI training or AI-answer citation is a genuine liability rather than a channel, that calculation flips, and blocking some or all of these bots is a reasonable, defensible choice – just decide per bot class instead of copying one blanket rule.
Whatever you decide, the mechanical checks stay the same as any other on-page change: confirm your robots.txt syntax is valid, verify the rules live on the actual served file rather than a cached copy, and if you add firewall rules, test that the blocked user-agent is actually refused rather than just logged. The broader on-page hygiene checks this site runs before any change ships are covered in the WordPress SEO checklist; if you’re also weighing an llms.txt file alongside robots.txt, that’s a related but separate decision – llms.txt is a proposed convention for pointing AI tools to clean content, not an access-control mechanism.
Frequently asked questions
Does blocking GPTBot in robots.txt stop ChatGPT from showing my page?
No. GPTBot only crawls for OpenAI’s model training data. ChatGPT’s live search and answer results come from OAI-SearchBot and, for user-triggered fetches, ChatGPT-User – both separate user-agents you’d need to block individually to affect what ChatGPT can show or cite.
Will a robots.txt rule actually stop every AI bot from fetching my page?
No, not fully. Most AI crawlers documented by their vendors do respect robots.txt, but OpenAI’s own documentation states ChatGPT-User does not, and Perplexity’s documentation says its user-triggered fetcher generally ignores robots.txt rules too. Stopping those specific fetches requires blocking the user-agent at the firewall or CDN level, not just in robots.txt.
Does blocking AI crawlers hurt my Google rankings?
Blocking AI-specific user-agents like GPTBot, ClaudeBot, or Google-Extended does not touch Googlebot, which is a separate crawler for Google Search. Google’s own documentation on Google-Extended states it does not impact a site’s inclusion or ranking in Google Search. The cost of blocking these bots is reduced visibility in AI training data or AI-assistant answers, not a Google Search penalty.
Leave a Reply