Crawler evidence on your site lives in three places: raw server logs, your CDN’s edge analytics, and whatever a WordPress plugin can parse from those logs. Google Analytics 4 is not one of them. GA4 fires through a tag that runs as JavaScript in a visitor’s browser, and every documented AI crawler – GPTBot, ClaudeBot, PerplexityBot, and the rest – fetches raw HTML without running that script, so none of their visits ever reach a GA4 report. Below is the current, vendor-documented roster of AI crawlers, where to actually find their hits, and what crawl frequency can and cannot tell you about whether an AI answer engine is likely to cite you.
Site owners keep asking a version of the same question: which AI crawlers are actually hitting my site, and how often. It is a fair question, and a different one from measuring whether you’re being cited in AI answers. Citation tracking asks what AI systems say about you. Crawler analytics asks what they’ve already read. Both matter, and neither substitutes for the other.
Why your analytics dashboard is blind to crawlers
GA4’s standard web measurement depends on the Google tag running client-side. Google’s own documentation on server-side tagging describes the default setup this way: without a server container in between, “the browser” sends “events directly to Google’s analytics servers.” That single sentence is the whole explanation. A hit only exists if a browser executes the tag and calls out to Google. AI crawlers, as documented by every vendor covered below, fetch pages to read their HTML content. None of the vendor documentation we could find describes any of these crawlers executing page JavaScript the way a browser does. No script execution means no gtag call, and no gtag call means GA4 has nothing to log. This isn’t a GA4 configuration mistake you can fix with a setting. It’s the measurement model working exactly as designed, for a use case – human browser sessions – that doesn’t include crawlers.
The current AI crawler roster
Four organizations publish documented, named user agents for content-related crawling. OpenAI documents four: GPTBot, used “to crawl content that may be used in training our generative AI foundation models,” which respects robots.txt; OAI-SearchBot, used “to surface websites in search results in ChatGPT’s search features,” also robots.txt-compliant; OAI-AdsBot, which only visits pages submitted as ads rather than crawling generally; and ChatGPT-User, which fires for live user actions inside ChatGPT and Custom GPTs – OpenAI states plainly that “because these actions are initiated by a user, robots.txt rules may not apply.”
Anthropic documents three: ClaudeBot, which “collects web content that could potentially contribute to” training Anthropic’s models; Claude-User, which fetches a page “when individuals ask questions to Claude”; and Claude-SearchBot, which “navigates the web to improve search result quality for users.” Anthropic states all three “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.”
Perplexity documents two, split the same way OpenAI splits theirs: PerplexityBot “surface[s] and link[s] websites in search results on Perplexity” and is not used for AI training, and Perplexity’s own docs recommend allowing it in robots.txt; Perplexity-User fires when “a user requested the fetch” and “generally ignores robots.txt rules” for that reason. Common Crawl’s CCBot rounds out the list – not an AI company itself, but the open dataset several foundation models are trained on, identifiable by the literal string “CCBot/2.0” and blockable with a two-line robots.txt rule.
The pattern across every vendor is the same split: a crawling bot that respects robots.txt and feeds training or search indexes, and a live-fetch bot that acts on a specific user’s request and may not check robots.txt at all, because the fetch was requested by a person, not scheduled by a crawler.
Reading server logs and CDN analytics for bot traffic
Server logs are the most complete record available, because every HTTP request lands in them regardless of whether it came from a browser, a script, or a crawler, and each entry carries the user-agent string these vendors document. The trade-off is access and tooling: raw logs require either shell access to grep them or a log-analysis tool pointed at them, and most shared WordPress hosts don’t expose raw logs by default.
CDN-level analytics sit a level up. Cloudflare, as one documented example, explicitly avoids collapsing everything into a single “AI bot” label – its own docs state that it “classifies bots by behavior,” using categories like search, agent, and training crawls, rather than one bucket. At the zone level, Cloudflare’s Bot Analytics (Business and Enterprise plans) shows traffic segmented by automated versus likely-automated type and the detection source behind that call; the Enterprise Bot Management tier adds a bot-score distribution across every request. Cloudflare also runs a public, aggregate dashboard – Radar’s AI Insights page – that tracks the top AI crawlers globally, but that’s industry-wide data, not your site’s own log.
WordPress plugins that surface “bot traffic” sit a layer above both: at best, they’re parsing the same access-log or CDN-log data described above and filtering it down to known user-agent strings, which is useful for a quick read but only as complete as the log source feeding it. None of that changes the underlying fact from the section above – whatever tool you use, you’re reading logs or edge data, never GA4.
What crawl frequency does and does not tell you about citation odds
It is tempting to treat crawl frequency as a leading indicator: more GPTBot hits should mean better odds of a ChatGPT citation. We looked for a published study or vendor statement tying crawl frequency to citation likelihood and found none. That gap is worth stating plainly rather than filling with a guess. What crawl logs can tell you, honestly, is narrower: whether a given crawler has access to a page at all, how recently it last fetched that page, and whether your robots.txt is blocking a crawler you didn’t mean to block. What they cannot tell you is whether that crawl turned into an answer, a citation, or nothing. That gap is exactly why crawler analytics and citation tracking are separate jobs, not one measurement done twice – our own citation-tracking method runs a query panel against live AI answers precisely because crawl logs can’t answer that question.
Turning the data into decisions: allow, block, or optimize
Once you can see who’s actually crawling, the decision splits into three options, not two. Allow a crawler when you want inclusion in that system’s answers or search index – PerplexityBot and OAI-SearchBot are both explicitly built for that and both vendors ask you to allow them. Block a crawler when you specifically object to your content training a model, understanding that a robots.txt rule only stops the crawlers that honor it; live-fetch bots like ChatGPT-User and Perplexity-User may bypass it entirely because a person, not a schedule, triggered the request. We’ve written the full mechanics of that trade-off, including working robots.txt lines for every crawler above, in our guide to blocking AI crawlers. The third option is easy to skip past: optimize instead of choosing allow-or-block at all, by making sure the pages you do allow are actually structured for a crawler to read cleanly, the same groundwork covered when we compared GEO, SEO, and AEO as distinct disciplines. This site runs that exact loop – observe crawl and GA4 data, propose a change, get human approval, apply it – as the same AI SEO agent process described elsewhere on this site, and we track our own referral side of this with AI referral traffic in GA4, which is the one piece of this picture GA4 actually can see, since a referral click comes from a browser.
Frequently asked questions
Why doesn’t Google Analytics show AI crawler visits?
GA4 measurement runs through a tag that executes as JavaScript in a visitor’s browser and then calls out to Google’s servers. AI crawlers fetch a page’s raw HTML without running that script, so no measurement call ever fires. GA4 can still show AI referral clicks, because those come from a real browser session, just not the crawl itself.
Which AI crawlers should I actually look for in my logs?
The vendor-documented list currently covers OpenAI’s GPTBot, OAI-SearchBot, OAI-AdsBot, and ChatGPT-User; Anthropic’s ClaudeBot, Claude-User, and Claude-SearchBot; Perplexity’s PerplexityBot and Perplexity-User; and Common Crawl’s CCBot, whose data feeds multiple foundation models.
Does more crawling mean a better chance of being cited?
There’s no published study or vendor statement tying crawl frequency to citation odds. Crawl logs can confirm a page was fetched and how recently; they can’t tell you whether that fetch produced an answer or a citation. Those are measured separately, through a citation-tracking method rather than a log.
Can a robots.txt rule stop every AI crawler?
No. Crawling bots like GPTBot, ClaudeBot, and PerplexityBot are documented as robots.txt-compliant. Live-fetch bots like ChatGPT-User and Perplexity-User are explicitly documented as sometimes ignoring robots.txt, because a specific user’s question triggered the fetch rather than a scheduled crawl.
Leave a Reply