.svg)
Incrementality is the share of a result that was actually caused by an action, versus what would have happened anyway. You measure it by comparing a treatment group against a held-out control: incremental lift equals treatment minus control. Without that counterfactual, you are crediting outcomes, not proving you caused them.
What is incrementality?
Incrementality answers a causal question — did this cause that? — while almost every analytics report answers an easier one: who gets the credit? Those are not the same question, and treating them as if they were is how teams end up confidently wrong. A dashboard can show that a conversion happened after someone saw your ad or used your feature. It cannot show that the conversion wouldn't have happened anyway.
The honest framing: most teams don't have an incrementality problem, they have a measurement-trust problem. They can produce a number for almost anything. What they can't do is defend it against the only question that matters — compared to what? That counterfactual — what would have happened with no action — is the whole game. Correlation says two things moved together. Causation says one moved the other. Incrementality is the discipline of telling them apart.
Most of this guide covers the broad, channel-aware version of that discipline, because it's what most searchers need first. But the version that actually decides roadmaps — product incrementality, where the thing you hold out is users rather than ad budget — is where the interesting failures hide. For the short definition on its own, see our glossary entry on incremental growth.
Incrementality vs attribution: why your last-click numbers lie
Attribution splits the credit for conversions that already happened. Incrementality asks the harder question: would those conversions have happened without you at all? One measures correlation across touchpoints; the other measures causation against a counterfactual. Last-click flatters branded search and retargeting because both intercept demand you already had.
Attribution is a credit-assignment system. It takes conversions that already happened and divides the trophy among the touchpoints that preceded them — last-click, first-click, time-decay, take your pick. Every one of those models shares the same fatal assumption: that the touchpoint caused the conversion rather than merely witnessing it. Branded search is the cleanest tell: someone has already decided to buy, types your name into Google, clicks the ad above the organic result, and last-click credits it for a sale it did nothing to create. Retargeting runs the same play on people already coming back.
The operator test cuts through it: switch the channel off and watch what conversions do — two weeks for an impulse buy, a full purchase cycle for a considered B2B sale. If they hold, you were paying to intercept demand you already owned. If they drop, you've found real lift. That blackout is a crude holdout — and a crude holdout usually beats a sophisticated attribution model, because one measures causation and the other only co-occurrence.
How to measure incrementality: the lift formula and a worked example
You measure incrementality by running the change against a randomly held-out control and subtracting one from the other. Treatment minus control is the incremental lift; that same difference over the control is lift percent. The control supplies the counterfactual, so whatever remains is the part you genuinely caused.
The formula
Incremental lift = treatment − control
Lift % = (treatment − control) / control
Incremental revenue = the revenue only the treatment produced (treatment revenue − control revenue). Incremental ROAS (iROAS) = incremental revenue ÷ ad spend.
Two definitions worth pinning down. Incremental revenue is the revenue only the treatment produced — treatment revenue minus what the control made anyway. Incremental ROAS (iROAS) is that figure over ad spend, almost always uglier than the ROAS your ad platform reports, because the platform counts conversions that would have landed without the ad.
Subtract the baseline before you celebrate.
A team ships a change and watches daily active users climb 12% week over week. The release gets the credit. But DAU was already growing about 7% on its own — seasonality, ongoing acquisition, the natural ramp of a new cohort. That 7% is the counterfactual: what would have happened anyway.
- Observed lift: 12%
- Organic baseline (what the held-out control did): ~7%
- True incremental lift: 12% − 7% = ~5%
More than half the celebrated number was the baseline the team forgot to subtract. The gap usually widens over time, because week-one figures are inflated by the novelty effect — users poke at anything new — which decays as behavior settles back toward baseline.
Made concrete: if treatment-group retention is 31% and the held-out control sits at 29%, incremental lift is 2 points, or (31 − 29) / 29 ≈ 6.9% — not the 31% the slide wanted to claim.
Here is the same logic on a live flow rather than a toy. When we ran a structured A/B program on the sign-up flow for HealthPlans of North Carolina, one change — a single plain-language line explaining why users had to enter their details at the final step — moved self-enrolment from 9.41% to 19.1%, roughly (19.1 − 9.41) / 9.41 ≈ 103% relative. A separate test surfacing sales agents on the homepage moved conversion 3.76% to 4.7%, about 25%. Same program, a near-doubling and a modest win — and only a controlled comparison against the unchanged flow lets you trust either number instead of crediting the month's traffic mix. (HealthPlans of North Carolina case study.)
Which is the rule that belongs on every results deck: a lift number without a confidence interval is a horoscope — specific enough to flatter the reader, and predictive of nothing.
Holdout tests, control groups, and geo experiments (the methods ladder)
A holdout test withholds the change from a randomly chosen control group, so the gap between the two groups is the causal effect — a randomized controlled trial applied to growth. When a holdout isn't possible, a ladder of weaker methods trades rigor for feasibility, and the rung you pick is a conscious choice.
Think of the methods as a ladder, ordered by how cleanly they isolate cause:
- Randomized holdout / RCT — withhold the change from a random control group. Causal by construction; take it whenever you can.
- Geo experiment (e.g. GeoLift) — when you can't randomize people, randomize markets: matched regions get the change, others don't.
- Feature A/B test — a randomized holdout at the level of a single product change.
- PSA / ghost ads / platform conversion lift — serve a placebo or hold back impressions to estimate ad lift where true randomization isn't yours to run.
- MMM (marketing mix modeling) — a top-down regression across all channels; triangulation, not ground truth, because it has no holdout.
The media-measurement world has written this ladder to death, so we'll keep it tight. If you're building the test itself, our A/B testing best practices guide covers the mechanics, and the experiment design template gives you a worksheet to scope one.
Product incrementality vs marketing incrementality (the part nobody else covers)
Marketing incrementality holds out a channel to prove ad spend caused conversions. Product incrementality holds out a cohort of users to prove that shipped changes — not seasonality or natural ramp — moved the north-star metric. The unit isn't a single feature; it's the cumulative effect of many small ships.
Here is the gap the ad-tech guides never close, because closing it isn't their job. Marketing has a clean unit: the campaign — hold out a channel, read the lift, decide the budget. Product teams don't ship campaigns. They ship a tooltip here, a reordered onboarding step there, a faster query, a new empty state — dozens of small changes in a quarter, almost none of them individually large enough to power a clean test on its own. So what actually happens? The team scans the dashboard each week, finds whatever moved, and pins it on whatever shipped nearby. At quarter's end the wins get summed into one triumphant slide.
That slide is fiction, and predictably so. Summing point estimates always inflates the total, for two compounding reasons: selection bias — you kept the experiments that looked good and quietly shelved the ones that didn't — and regression to the mean, because the metrics you celebrated were disproportionately the ones having a lucky week. Add dozens of noisy estimates, keep the flattering ones, and you've manufactured growth the north-star metric never actually saw.
The fix is to change the unit of measurement. Instead of asking did this one feature work, hold a cohort of users out of the cumulative stream of changes and read the north-star against them over a long window — a long-run product holdout. A small, randomly chosen slice of users stays on a frozen baseline experience for a quarter or two while everyone else gets the full firehose of ships. The gap between the two groups at the end is not the effect of any single feature; it is the honest, compounded effect of the entire quarter of product work — the cleanest number that answers did all of this move the metric, or would users have gotten there anyway? For one high-stakes change on its own, a feature holdout does the same job at smaller scale. And mind the novelty effect while you're at it: anything new gets poked at simply because it's new, which inflates early reads and then fades — a real effect is still there once the novelty audience has cycled through; a fake one is gone within a few weeks.
This is also where measuring incrementality and doing marketing part ways. When we ran a structured experimentation program for Komfortkissen — six sequential controlled experiments over three months — the flagship experiment alone delivered a +7.5% overall revenue uplift. The senior read isn't that headline, though. The winning benefit-first image test reported +13.9% conversion lift at 89% odds-to-win — a deliberate bet, not a 95% certainty — and +8.3% more revenue per visitor before it shipped; a segmentation cut showed returning users converted 29% better with a dropdown size selector while new users preferred chips, an effect the average hid completely; and an on-brand orange CTA came in at −7.93% and was killed, not buried in an appendix. A program that reports its losers and its segment splits is measuring incrementality; one that only reports the +7.5% is doing marketing. (Komfortkissen CRO & experimentation program.)
You can't measure incrementality on a broken data foundation
Incrementality is a subtraction between two numbers, so it inherits every flaw in how those numbers are produced. If events double-fire, conversion definitions drift across platforms, or IDs don't stitch, the noise swamps the real lift, and your "effect" is an instrumentation artifact. The readiness check comes before the experiment.
This is the prerequisite the vendor guides skip, because admitting it doesn't sell a dashboard. You cannot measure incrementality on a broken data foundation — and "broken" is more common than anyone admits. Those failure modes aren't edge cases; they're the default state of most stacks, and each one quietly turns the difference you compute into an artifact of instrumentation rather than a fact about your users.
The order of operations is the whole point — sequence the audit ahead of the test, not after it quietly fails. Before you run a holdout, confirm the events fire once, the conversion definition is identical on both sides, and identity stitches across surfaces. It's the unglamorous part that decides whether the lift number means anything. If you're not sure your tracking can support a clean read, an experimentation readiness audit surfaces the issues that would invalidate your results, and a data foundation program — or a focused data stack audit — fixes them at the root.
When you can't run a clean experiment: the rigor ladder
When you can't randomize — a pricing change, a rebrand, a 200-account enterprise motion — you don't get to fake certainty. You climb down a rigor ladder from randomized holdout to modeled pre/post, take the strongest rung your situation allows, and label exactly where you landed so nobody downstream over-claims later.
The grown-up answer to what do you do when a clean experiment is impossible is not to invent one — and not to throw up your hands either. Every rung below gives you a real, if weaker, causal read; the skill is knowing which one you're standing on and saying so out loud, so the next person doesn't read a rung-six estimate as gospel. Strongest to weakest:
- Randomized holdout / RCT — random assignment to treatment and control. Causal by construction. Take it whenever you can.
- Geo experiment / GeoLift — randomize markets when you can't randomize people. Strong, with more noise and fewer units to work with.
- Difference-in-differences vs a matched control — compare the treated group's before/after change to an untreated group's, netting out shared trends.
- Synthetic control / CausalImpact — build a weighted "synthetic" control from untreated units that tracked you before the change, then read the divergence after.
- Propensity matching — construct a comparable control by matching on observed characteristics. Only as good as the variables you actually have.
- MMM as triangulation — a top-down model across channels. A cross-check, never a verdict on its own.
- Pre/post with a modeled baseline — compare after to a modeled estimate of what would have happened. The weakest rung, and by far the most common.
The discipline is the label, not the ladder itself. What you may not claim: anything on rungs six and seven is a directional read and a hypothesis, not proven causation. You can use it to decide where to aim a real experiment next; you cannot use it to tell the board you caused the number. Blurring that line is how "data-driven" decisions go quietly, expensively wrong.
Statistical rigor: power, duration, and what a good lift actually looks like
There's no universal "good" lift number. A good lift is one whose confidence interval clears zero at your traffic volume — so compute the minimum detectable effect before you run, not after. The interval, not the point estimate, tells you whether the result is real or noise dressed up as a win.
"Good" is a function of your traffic, not a number you can borrow from a blog. The minimum detectable effect (MDE) is a planning input, not an afterthought: before you run, work out the smallest lift your sample could detect at your conversion rate — if that floor sits above the lift you realistically expect, the test is dead on arrival. A rough floor, once you've run that math: about four weeks and at least ~200 conversions in the control arm for funnel tests — though the honest target is whatever your MDE demands. Product holdouts measuring retention need a quarter or more, because compounding shows up in later curves, not week one.
On what a "good" lift looks like, treat these as rough, directional industry ranges — not measured guarantees: lower-funnel and performance changes often land around 15–30%, prospecting can show 30–100%+, and retargeting often only 5–20% — that low end being the tell that retargeting mostly re-bills demand that existed. The real decision rule isn't the headline anyway; it's the interval. A point estimate of +3% sitting at [−4%, +10%] and the same +3% at [+1.5%, +4.5%] are opposite ship decisions: the first might genuinely be zero, the second is reliably positive. Read the interval first. The mechanics — p-values, intervals, sample size — are in our A/B testing statistics explained guide.
Where Adasight fits
The through-line is simple: incrementality is a subtraction between two numbers, so the answer is only as trustworthy as the data foundation beneath it and the experiment design around it — the gap we close for product and growth teams. If you suspect your tracking can't support a clean read, start with a readiness audit, not a test. If your data is sound but you're still summing pre/post point-wins, the move is a properly powered, controlled program — an experimentation growth engine that kills its losers. Fix the foundation first and the lift number stops being a talking point and becomes a decision you can bank. Happy to put a second set of eyes on whether your setup supports a clean read.
If your team has ever shipped on a number that didn't hold up, the A/B Testing Playbook — how to stop calling false winners walks through how to stop. Grab it before your next launch review.
Frequently asked questions
What is incrementality in simple terms?
Incrementality is the part of a result your action actually caused, versus what would have happened anyway. You isolate it by comparing a treatment group against a held-out control: incremental lift = treatment − control. Without that comparison, you're crediting outcomes, not proving you caused them.
What is the difference between incrementality and attribution?
Attribution divides credit for conversions that already happened; incrementality asks whether they would have happened without you. Attribution measures correlation, incrementality measures causation. Last-click routinely over-credits branded search and retargeting because they intercept demand that already existed. The test: if you paused the channel for two weeks, would conversions actually fall?
How do you calculate incremental lift?
Incremental lift = treatment − control. Lift % = (treatment − control) / control. If your treatment group converts at 31% and a held-out control converts at 29%, the incremental lift is 2 points, or (31−29)/29 ≈ 6.9% — not the full 31%, most of which would have happened anyway.
What is a good incremental lift percentage?
There's no universal number — a good lift is one whose confidence interval clears zero for your traffic. As rough, directional benchmarks: ~15–30% for lower-funnel/performance, ~30–100%+ for prospecting, and only ~5–20% for retargeting, because retargeting mostly captures demand that already existed.
What is a holdout test and how does it work?
A holdout test deliberately withholds the change (a feature, an ad, a flow) from a randomly selected control group, so the difference between the two groups is the causal effect. A clean holdout is a randomized controlled trial applied to growth — the strongest way to measure true incrementality.
How long should an incrementality holdout test run?
Long enough to clear the novelty effect and reach statistical power. A practical floor is about four weeks and at least ~200 conversions in the control arm; product holdouts measuring retention or cumulative impact should run a quarter or more, because compounding effects show up in later retention curves, not week one.
Can you measure incrementality without running an experiment?
Partly. When you can't randomize, you climb down a rigor ladder and label where you land: geo experiment, difference-in-differences, synthetic control, MMM as triangulation, then pre/post against a modeled baseline. Each step trades certainty for feasibility — and on the weaker rungs you report a direction, not proven causation.
What is the difference between MMM and incrementality testing?
MMM (marketing mix modeling) models all channels at once without a holdout — it's correlational. Incrementality experiments are the ground truth that keeps it honest: calibrate the model against real holdout tests on channels you can hold out, and treat any MMM never validated by an experiment as a hypothesis, not a measurement.
What is the difference between product and marketing incrementality?
Marketing incrementality holds out a channel to prove ad spend caused conversions. Product incrementality holds out a cohort of users to prove shipped product changes caused real movement in the north-star metric. The product unit isn't the campaign — it's the cumulative effect of many small ships, best measured by a long-run product holdout.
What is incremental ROAS (iROAS)?
Incremental ROAS is incremental revenue divided by ad spend — the return from revenue your spend actually caused, not all revenue last-click credited to it. It's almost always lower than reported ROAS, because standard attribution counts conversions that would have happened without the ad.
Is an A/B test an incrementality test?
Yes, at the feature level. A clean A/B test randomly assigns users to a treatment (new experience) or control (old experience), so the difference between them is the incremental effect of that one change. The discipline of incrementality is the same — randomized control group, isolated causal read.
What is the novelty effect?
The novelty effect is the temporary behavior bump a change gets simply because it's new — users poke at anything unfamiliar. It decays. Read a holdout at day three and you'll over-state lift; if the effect is real it should still be there at day 21, after the novelty audience has cycled through.
What is a counterfactual or baseline?
The counterfactual (or baseline) is what would have happened without your action — the number the held-out control group reveals. Incrementality is the gap between what actually happened and that counterfactual. Forget to subtract it and you'll credit your change with growth that was already underway.
