In 2019, Marketing Science published what I consider the single most important piece of empirical research on digital advertising measurement of the last decade, and it came from an unexpected direction. Brett Gordon, Florian Zettelmeyer, Neha Bhargava, and Dan Chapsky, working with Facebook’s own advertising experimentation infrastructure, took 15 large-scale U.S. advertising experiments, 500 million user-experiment observations, 1.6 billion ad impressions, and asked the question the industry had spent 15 years avoiding. Treat the randomized experiment as ground truth. How close do the standard observational approaches come to recovering it?
Not close. And not predictably. The observational methods under test weren’t straw men. Exact matching, propensity score matching, regression adjustment... the respectable machinery of applied causal inference, running on user-level data richer than most advertisers will ever possess. Against the experimental benchmark, these methods missed widely, frequently overestimating incremental conversions by a factor of 3 or more, sometimes underestimating, with no reliable way to know in advance which direction or by how much. The richness of the data didn’t save the estimates, because data volume was never the problem. Ad delivery systems target people who are already more likely to convert, and no amount of adjustment on observables fully removes a selection process operating on unobservables. Gordon, Robert Moakler, and Zettelmeyer replicated the exercise across hundreds of experiments in 2023, again in Marketing Science. Same conclusion, larger scale.
The industry absorbed all this as a vague sense that attribution is imperfect. That’s a misreading. The research says something sharper and more useful: platforms are capable of producing 2 entirely different classes of measurement, and only one of them is evidence.
The 2 Products in the Same Dashboard
The first class is the genuine randomized lift study. The major platforms operate experimentation systems that can randomly assign eligible users to a treatment group served the campaign and a control group that isn’t, then compare outcomes. When the randomization is real, this design inherits the full authority of the randomized controlled trial. The refinements documented in the academic literature, including the ghost ads approach published in the Journal of Marketing Research in 2017, which identifies the control-group users who would have been served the ad, exist precisely to make platform experiments economical and clean. A well-run platform lift study is, within its scope, the best evidence money can buy about that platform’s causal effect.
The second class is modeled and attributed measurement. Conversion attribution windows. Modeled conversions filling privacy gaps. Algorithmic lift estimates produced without randomized holdouts. The estimated performance figures populating default reporting. These numbers come from exactly the observational machinery the Gordon studies benchmarked, or from machinery with even less identification behind it. They’re operationally useful for running campaigns. They aren’t causal evidence, and the 2019 results quantify how far from evidence they can be while looking entirely reasonable on a screen.
Here’s the structural problem. Both classes arrive through the same interfaces, with similar precision, from the same vendor, and the ambiguity serves the vendor. A platform that sells media and also supplies the measurement of that media carries a conflict of interest that no individual employee’s integrity resolves, because the conflict lives in the product design. Which numbers get foregrounded. Which studies are easy to run. Which results get framed as performance. The eBay episode from earlier in this series, where genuinely randomized geographic holdouts revealed near-zero incrementality that attributed reporting had valued in the tens of millions, is what it looks like when an advertiser finally separates the classes for themselves.
And the blurring has gotten worse since 2019, not better. Privacy changes, identifier deprecation, and operating-system consent frameworks removed a large share of observed conversions, and the platforms filled the gap with modeled ones: estimated events standing in for unobserved events. Modeled conversions now flow into the same reporting surfaces, and in some configurations into the outcome metrics of studies presented as experiments. Which means an advertiser can be shown a randomized design whose outcome variable is partly synthetic. The randomization can be genuine while the measurement of the outcome is estimated, and the resulting number sits in an uncomfortable middle category the dashboard doesn’t label. So the classification question no longer stops at “was assignment randomized.” It extends to the outcome side. What fraction of the counted conversions were observed, what fraction were modeled, and does the read change if the modeled fraction is excluded? I ask this in vendor reviews. The answer changes the conversation, and occasionally it changes the result.
The 3 Questions That Separate Evidence From Reporting
The practical leadership skill is interrogation. It takes exactly 3 questions, and none requires a statistics degree.
Was assignment randomized, and by what mechanism? A true lift study answers concretely: eligible users were split before the auction, the control group was withheld from delivery, the mechanism is documented. A modeled lift number can’t answer it, and the vendor’s representative will usually respond with methodology language describing adjustment rather than randomization. The distinction is binary. The answer is yes with a mechanism, or it’s no.
Who holds the holdout, and can it be audited? Randomization run entirely inside the platform, measured entirely by the platform, reported entirely by the platform is real evidence with a custody problem. The mature versions of this arrangement include third-party measurement partners, exportable experiment-level results, and designs the advertiser co-specifies, power calculations agreed in advance. The economics literature is blunt that advertising effects are small relative to outcome variance, which means underpowered lift studies produce noise wearing error bars, and a platform that runs 10 underpowered studies for you will eventually hand you an impressive result by chance. Study registration and pre-agreed power are the guardrails. They exist only if the advertiser demands them.
Does the outcome metric reconcile to anything we recognize? A lift study measured on platform-defined conversions inherits every definitional gap between the platform’s event and your ledger. This is the same discipline I wrote about in August. A lift result that can’t be walked, even approximately, to a financial quantity will win the meeting and lose the CFO. The strongest programs pipe experiment outcomes into their own measurement stack, calibrate the MMM against the experimental reads, and reconcile the combined picture to financial actuals, so the platform experiment becomes one anchored input instead of a free-floating claim.
Where This Leaves the Leader
I run measurement for a $1B portfolio, and my posture toward platform measurement is classification. Not trust. Not boycott. Every platform number entering our decision processes gets classified at intake as experimental or observational, the way a bank classifies collateral. The categories are written down, and a new source gets classified before its numbers circulate. Experimental results with documented randomization, adequate power, and reconcilable outcomes get treated as evidence and used to calibrate models. Everything else gets treated as operational telemetry... useful for running campaigns, inadmissible for judging them.
That posture also resolves the apparent paradox in the Gordon research. The same company whose default reporting embodies the observational methods that missed truth by multiples also built the experimentation infrastructure that established the truth in the first place. Both facts are permanent. The platforms will keep selling modeled confidence because it scales, and they’ll keep operating genuine experimentation systems because sophisticated advertisers demand them. Which product your organization consumes was never the platform’s decision. It’s yours, and it shows in whether your team can answer the 3 questions for every lift number that reached a budget conversation this year.
The 2019 study has been public for 6 years. The 2023 replication removed the remaining excuses. Organizations still allocating on modeled lift aren’t uninformed. They’re unwilling. The distance between those two words is the entire distance between a measurement program and a reporting habit.
Every lift study I have run says the same thing: most reported performance is demand that already existed.