Three named bots account for most of the AI traffic hitting your server: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and its siblings (Anthropic), and PerplexityBot (Perplexity). All three respect robots.txt for their automatic crawling, so you can allow or block each one by name in a single file. What you can’t easily get back is a referral click – this site’s own analytics show zero sessions from any AI assistant in the last hundred days, even though these bots are documented as actively crawling the web.
I went to each platform’s own documentation rather than a summary of it, because the summaries I found disagreed with each other on small but important details – which bots respect robots.txt, which don’t, and why. Below is what the primary sources actually say, what independent, directly-fetched evidence adds on top of that, and what I’m choosing not to claim because I couldn’t verify it myself.
What platform docs actually say
OpenAI runs three separate bots with three separate jobs. OpenAI’s own bot documentation describes GPTBot as the one that “crawls content for training OpenAI’s generative AI foundation models” – disallowing it signals that your content shouldn’t be used in model training. OAI-SearchBot is different: it surfaces websites in ChatGPT’s search features, so blocking it means your pages won’t show up in those search answers. ChatGPT-User is different again – it fires when a person inside ChatGPT asks the model to fetch a specific page, not as part of a bulk crawl, so robots.txt rules may not apply to it the same way they apply to the other two. All three identify themselves by name in their user-agent string, and GPTBot’s IP ranges are published at openai.com/gptbot.json if you want to verify a request instead of trusting the header.
Anthropic’s documentation describes a similar three-way split. Its support article on Claude’s bots says ClaudeBot “gathers public web content to enhance AI model training and safety,” Claude-User fetches a page only when a user directs Claude to do so, and Claude-SearchBot indexes content “to improve search result quality and relevance for users.” The documentation states plainly that “all three bots respect industry-standard ‘do not crawl’ directives in robots.txt files,” and Anthropic also supports the non-standard Crawl-delay directive if you want to slow it down rather than block it outright. One caveat worth repeating in Anthropic’s own words: blocking the bots’ IP addresses directly “may not work correctly or persistently guarantee an opt-out,” because that can stop the bot from reading your robots.txt file in the first place – the file-based block is the one the documentation actually recommends.
Perplexity draws the clearest line between its two bots. Perplexity’s own guide says PerplexityBot “is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models,” and the documentation recommends allowing it. Perplexity-User is the other half: it runs when a Perplexity user’s question needs a specific page fetched in real time, and because the user – not the bot – initiated that fetch, Perplexity’s own guide says it “generally ignores robots.txt rules.” That’s stated directly by Perplexity, not inferred by me.
Put together, the pattern across all three platforms is the same shape: one bot for training, one bot for search indexing, and one “user-fetch” bot that behaves differently because a person asked for it in the moment. If you only read about “the AI crawler” as a single thing, you’re missing that the three jobs carry different rules, and blocking the wrong one can cost you more than you intended – this is a different problem than classic SEO versus AEO positioning, since it’s about access, not ranking.
What the independent evidence supports
Platform documentation tells you what a bot is supposed to do. It doesn’t tell you how much of it is actually happening, so I went to Cloudflare’s own data, since Cloudflare sits in front of enough of the web to measure crawler traffic directly rather than estimate it.
Cloudflare’s own 2025 crawler report found that GPTBot’s share of AI-crawler traffic rose from 5% in May 2024 to 30% in May 2025, with raw GPTBot requests up 305% over that window. ClaudeBot moved the other direction on share – down from 27% to 21% over the same period – even though Cloudflare’s data shows its absolute request volume still grew elsewhere in the same year. Share and raw volume are two different measurements, and I’m keeping them separate here rather than letting a falling share imply a falling total, because the report doesn’t support that implication.
The more pointed finding is about what these bots give back. In its 2025 year-in-review, Cloudflare states directly that “Anthropic had the highest crawl-to-refer ratio among the leading AI and search platforms” in 2025. I’m reporting that ranking claim and nothing more specific – several widely-shared numeric ratios (500,000:1 and similar figures) are circulating in third-party summaries, but I couldn’t confirm an exact number on Cloudflare’s own page, so I’m not repeating a number I can’t trace back to its source. The qualitative claim – Anthropic crawls disproportionately more than it refers visitors back – is the one I can stand behind.
I also checked this site’s own numbers rather than assume the industry pattern applies here. Over the trailing roughly hundred days, this site’s GA4 data shows zero sessions arriving from chatgpt.com, claude.ai, perplexity.ai, or any comparable AI-assistant referral source. The actual traffic in that window was direct visits, a handful from Google organic search, and two small referral sources unrelated to AI assistants. These bots may well be crawling this site – I haven’t independently measured that – but none of them have sent a reader here yet. I’d rather publish that zero than imply a result I don’t have.
A practical checklist
Given only what’s verifiable above, here’s what I’d actually do about it:
- Decide per bot, not per platform. GPTBot trains models, OAI-SearchBot feeds ChatGPT’s search results, and ChatGPT-User fetches pages a person asked for – blocking all three with one rule can remove you from search answers you actually wanted to appear in. The same split applies to Anthropic’s and Perplexity’s bot families.
- Write the rules in your site’s robots.txt rather than relying on an IP block. Anthropic’s own documentation warns that blocking IPs can interfere with the bot reading robots.txt in the first place, which defeats the purpose. Pair it with an llms.txt file if you want to be explicit about what you’d prefer these bots do with your content, though none of the three platforms document llms.txt as something they currently honor.
- If you want a quick view of who’s actually requesting pages before you write any rule, check your own server logs for AI crawler activity rather than assume the published market-share figures apply to your traffic specifically.
- If you’ve decided to block broadly, understand what blocking actually costs you before you do it – being excluded from training is not the same trade-off as being excluded from a platform’s live search answers.
- Don’t expect a referral click as the reward for being crawled. Cloudflare’s own data points at a crawl-heavy, refer-light pattern industry-wide, and this site’s own zero is consistent with that. Measure citations and mentions on their own terms instead of waiting for the kind of traffic classic SEO trained you to expect.
Honest limits of current evidence
None of the platform documentation I read publishes a number for how often these bots actually visit a given site, so I’m not going to invent a crawl-frequency figure or a server-load estimate – the checklist above describes mechanisms you can act on, not a magnitude I can’t verify. Cloudflare’s market-share and crawl-to-refer figures are also aggregate, industry-wide numbers; they describe the web Cloudflare sees, not necessarily what’s happening on any one site, including this one. And the “zero AI referrals” figure I reported for this site is a hundred-day window on a newer site with modest traffic – it’s a real measurement, not a permanent verdict, and I’ll revisit it as more data accumulates.
The short version, restated: three platforms, three documented bot families, and a consistent robots.txt-based control point across all of them. What’s genuinely uncertain is the payoff – the independent evidence available right now points toward these bots taking more than they send back, and this site’s own data hasn’t shown a single referral yet. I’d rather you plan around that honestly than around a number that sounded better but didn’t come from anywhere I could check.
Leave a Reply