Separating Content Impact from Platform Growth in ChatGPT Referrals

When ChatGPT referral traffic doubles following a content update, the increase in visits is clear. What remains unclear is how much of that growth the project actually caused.

Several external factors can drive referral spikes independently of on-page changes. Overall ChatGPT adoption might be rising, the platform might be linking out more frequently, or the underlying topic might be gaining popularity. If untouched pages on the same site also see more referrals, a simple before-and-after comparison may overstate the impact of the work.

A recent study from Glasp attempts to isolate this effect using first-party analytics. The authors compared a group of updated pages against the remaining, untouched pages on the same domain. While this approach is much stronger than looking at raw growth multiples in isolation, the methodology still leaves significant room for doubt.

The value and limits of a same-domain comparison

Starting in January 2026, Glasp rolled out a set of updates to its YouTube Q&A pages. To measure the impact, the researchers tracked ChatGPT referral traffic to these specific pages and compared it against traffic to the rest of glasp.co.

Using the rest of the site as a baseline has clear advantages. Both groups of pages share the same domain authority, analytics setup, and exposure to broader shifts in how ChatGPT sends traffic. Tracking them side-by-side helps separate sitewide or platform-wide tailwinds from growth specific to the updated pages.

The study also benefits from a concrete outcome metric: actual recorded sessions where the source contained "chatgpt." Unlike abstract citation or visibility scores, this measures real visits captured by analytics—though it naturally misses users who were influenced by an AI answer without clicking a link.

However, this was not a randomized experiment. The updated Q&A pages and the remaining site pages served different user intents, contained different types of content, and were not matched for equivalence beforehand. Sharing a root domain does not make two sets of pages a true apples-to-apples comparison.

Why the adjusted estimate remains uncertain

The researchers analyzed 47 weeks of data—26 weeks prior to the updates and 21 weeks after. Depending on the statistical model used, they estimated that the updates increased referral traffic by a factor of roughly 1.8 to 2.3 (with one primary specification landing at 1.82x).

It is important to read this as a modeled observational estimate, not a guaranteed multiplier that anyone can unlock by copying the same tactics.

Two details in the paper highlight why caution is necessary. First, the ratio of traffic between the updated pages and the control pages already showed a significant trend before the changes were ever made. Because the two groups were not moving in parallel beforehand, the final estimate relies heavily on whether the statistical model accurately guessed how that pre-existing trend would have continued on its own.

Second, a placebo-in-time test yielded a p-value of 0.16. This check works by testing fake intervention dates to see if the actual launch date stands out as uniquely impactful. A result of 0.16 falls short of the standard 0.05 threshold for statistical significance. While this does not prove the updates had zero effect, it makes it harder to argue with confidence that the January rollout specifically caused the shift.

Ultimately, the paper provides suggestive evidence and a thoughtful framework for accounting for background growth. Still, the diagnostic checks do not support treating the causal effect as a settled fact.

How control groups can muddy the waters

Relying on an internal control group introduces its own set of complications. For example, if updates to one section of a site improve how AI systems view the domain as a whole, that positive spillover could boost the untreated pages too. That would shrink the gap between the two groups, though it does not automatically mean the final estimate is conservative.

Other unmeasured differences can skew the numbers in the opposite direction. The two sets of pages might face entirely different seasonal demand, or they might be affected by unrelated site changes. Notably, a change to bot-filtering rules in mid-March altered how sessions were recorded right in the middle of the study window, introducing measurement noise.

Finally, because Glasp applied multiple changes at once, the study cannot isolate which specific tactic actually moved the needle. A positive result for a bundled update does not validate every individual item inside the bundle.

How to evaluate referral-growth claims

When reviewing any claim about AI referral growth, start by looking at the raw baseline numbers and the timeline. A massive percentage jump is less impressive if it starts from a handful of visits, and shifts in analytics definitions or bot filtering can easily mimic real growth.

Next, look at how the baseline comparison was constructed:

• Did untouched pages on the site grow during the same window?
• Were the treated and untreated pages trending along the same path before the project started?
• Why were those specific control pages chosen, and is there any chance the project affected them too?

Finally, keep the metric tied to the actual goal. If a project aims to drive traffic, look at session counts; if the goal is qualified leads, evaluate conversions directly. Improvements in AI citations may be worth tracking, but they are not a substitute for measuring visits or business outcomes.

Adding a control group asks a better question, but it does not automatically prove causation. The most credible analyses show how the numbers shift under different assumptions—and are upfront about the uncertainty that remains when the data cannot give a definitive answer.