Perplexity publishes detailed documentation about what its crawlers do: PerplexityBot indexes pages for search results, and a separate fetcher called Perplexity-User retrieves a page only when a user asks a direct question about it. What Perplexity does not document is how it ranks the pages it retrieves – its Search API guide describes filtering and content controls, not a ranking method. Independent evidence adds a live, unresolved dispute on top of that documentation gap: Cloudflare says Perplexity crawled disallowed sites using an undeclared, browser-mimicking identity in 2025, and Perplexity has publicly rejected that framing. On this site, referrals from perplexity.ai have been zero in every check I’ve run so far, and I’m reporting that number as it stands rather than waiting for a better one.
What Perplexity’s documentation actually says
Start with the part Perplexity does explain. Its Search API documentation describes an endpoint that returns real-time, ranked web results as structured data, and it documents the controls you’d expect: you can filter by domain, restrict by language or region, and tune how much retrieved content the model gets to work with via context-size settings. What the same page does not do is explain the ranking technique behind those results. There’s no published formula, no disclosed weighting of freshness versus authority versus relevance – just the fact that results come back ranked, without a description of how.
The crawler documentation is more specific, and worth reading carefully because it draws a distinction most SEO advice glosses over. Perplexity’s own crawler resource page states that PerplexityBot exists to “surface and link websites in search results on Perplexity” and explicitly says it “is not used to crawl content for AI foundation models.” The same page recommends allowing PerplexityBot in robots.txt if you want to appear in Perplexity’s results, though it stops short of an explicit promise that PerplexityBot will always obey a disallow rule – it’s a recommendation to allow-list, not a compliance guarantee spelled out in those terms.
Perplexity-User is a different animal entirely, and the documentation is unusually direct about it. It’s described as supporting “user actions within Perplexity” rather than search indexing, and the docs state plainly that “since a user requested the fetch, this fetcher generally ignores robots.txt rules.” That’s Perplexity’s own language, not a third-party inference: a live, user-triggered fetch is treated differently from a standing crawl, and robots.txt is not expected to stop it. Perplexity also publishes IP ranges and exact user-agent strings for both PerplexityBot and Perplexity-User so site owners can verify traffic against something more reliable than a user-agent header alone, which is the same kind of verification approach I’ve recommended when writing about what an llms.txt file can and can’t control on a WordPress site – a directive is only as good as your ability to confirm who’s actually reading it.
What the independent evidence supports
Outside Perplexity’s own documentation, the most concrete recent evidence is a dispute, not a settled fact, and I want to present it as one. In August 2025, Cloudflare published a post alleging that Perplexity used an undeclared, Chrome-impersonating user agent and rotated across different network ranges to keep fetching content from domains that had explicitly disallowed its declared crawler in robots.txt – Cloudflare’s own words describe this happening “across tens of thousands of domains and millions of requests per day.” That’s a serious, specific accusation from a company with visibility into a large share of global web traffic, and it names exact behavior rather than gesturing at a vague concern.
Perplexity disputed the characterization publicly, with the company’s response – reported by outlets covering the exchange – describing Cloudflare’s post as closer to a sales pitch or a publicity move than a technical finding. I’m not going to adjudicate that argument here; both sides have an incentive in how this story gets told, Cloudflare sells crawler-management tools and Perplexity sells an AI search product. What I can say with confidence is that this wasn’t the first time Perplexity’s crawling behavior drew scrutiny – an earlier investigation in June 2024 reported the company accessing a test site from unpublicized IP addresses despite a robots.txt disallow rule, which means the current dispute sits on top of an existing pattern of questions rather than arriving out of nowhere. If you’re deciding how much weight to put on any single crawler’s stated intentions, that history is worth knowing, in the same way I’ve argued that overlapping AI-search terminology deserves scrutiny in GEO versus SEO versus AEO – a label is not the same thing as a verified behavior.
A practical checklist
Given what’s actually documented versus what’s disputed, here’s where I’d focus effort:
- Allow-list PerplexityBot in robots.txt if you want your pages eligible for Perplexity’s search results at all – the documentation recommends this explicitly.
- Don’t assume an allow rule for PerplexityBot changes anything about Perplexity-User’s behavior; the docs describe the two as separate fetchers with separate purposes.
- Verify crawler traffic against Perplexity’s published IP ranges and user-agent strings rather than trusting the user-agent header alone, especially given the Cloudflare allegations about undeclared identities.
- Don’t treat “ranked web results” in the Search API docs as a description of an SEO ranking algorithm – it describes an output format, not a disclosed methodology you can optimize against.
- Read the Cloudflare-versus-Perplexity dispute as two companies with conflicting incentives making conflicting claims, not as a resolved technical verdict either way.
- If you’re building a broader AI-citation strategy, treat structural fixes like clean answer formatting and clear sourcing as more durable than reacting to any single crawler controversy – the mechanics I cover in how to get cited by ChatGPT and Perplexity hold regardless of how this particular dispute resolves.
Honest limits of current evidence
I don’t have a ranking formula to give you, because Perplexity hasn’t published one, and I’m not going to invent one dressed up as an estimate. I also don’t have an independent, large-sample study of what actually predicts citation frequency on Perplexity specifically, the way some researchers have attempted for ChatGPT – if that research exists and holds up, I haven’t found it verified enough to cite here. That’s a real gap, not a number I’m rounding down to zero for effect.
What I do have is this site’s own data, and I’m reporting it as-is: across the GA4 referral pulls I’ve run recently, sessions arriving from perplexity.ai have been zero every time, out of a small total session count overall. That’s a small sample from one site, not evidence about Perplexity broadly, and it may simply reflect that this site doesn’t yet have enough total traffic or enough Perplexity-relevant content for a citation to show up as a referral. I’m not framing zero as proof of anything beyond what it is – the current measured state of one site’s own numbers. The same caution applies to structured-data efforts more broadly, which is why I’ve been careful to describe schema’s effect on AI citations as still evidence-thin rather than a settled win when I’ve looked at it directly. When this site’s Perplexity referral count changes, up or down, I’ll report the new number here.
Leave a Reply