.svg)
How to Measure Impact When You Can't Run a Clean A/B Test
At our recent webinar, Adasight x Cherto: 5 Lessons Learned from 1,000s of Experiments, an attendee asked Dr. Simon Jackson a question every team eventually hits: what do you do when a clean A/B test simply isn't possible? His answer was refreshingly unglamorous — you triangulate. Rather than force a rigorous test the data can't support, you gather several imperfect signals and synthesize across them. This post breaks that approach down into something you can actually run.
Want to watch the webinar for free?
The short version: When you can't run a clean A/B test — because you can't randomize, don't have the traffic, or the change already shipped — don't force one with heavy statistics. Triangulate instead: gather several independent, imperfect signals (a lightweight or partial A/B test, before-and-after data, product analytics, and qualitative research) and see whether they converge on the same story. No single source proves causation, but when they all point the same way, you have a credible, defensible read — reported as a direction and a confidence level, not false precision.
First: why you sometimes can't run a clean test
A "clean" A/B test needs two things — the ability to randomly split users, and enough of them to reach statistical significance. Plenty of real situations break one or both:
- Not enough traffic. Low-traffic pages, early-stage products, and most B2B tools simply can't reach a significant sample in a reasonable timeframe. The test would run for months — and by then the context has changed. (This is a statistical power problem more than a tooling one.)
- You can't randomize. Some changes hit everyone at once by nature — a pricing change, a rebrand, a new policy, a single big launch. There's no held-back control group to compare against.
- The change already shipped. By the time you're asked "did that work?", the change is live for 100% of users and no control was ever kept.
- Ethical or practical constraints. Sometimes you can't justify withholding an improvement from half your users, or the engineering to gate it isn't worth it.
- Spillover between groups. In products with network effects, the control group gets "contaminated" by the treatment (users talk, share, interact), so even a randomized split isn't clean.
If any of these sound familiar, you're not doing experimentation wrong — you're in the majority of real-world situations.
The shift: from one clean test to triangulation
Triangulation borrows its logic from navigation and detective work: no single bearing (or witness) is conclusive, but several independent ones that agree pin down the truth. Applied to impact measurement, it means you stop hunting for one perfect causal number and instead assemble multiple independent, imperfect signals. Each has a different weakness — so when they disagree, you learn something, and when they agree, the odds that they're all wrong in the same direction get small.
The trade is honest: you give up a single precise causal figure and gain a defensible direction with a stated confidence level. As Simon put it, it's not the most satisfying answer — but in these scenarios it's usually the right one.
The four evidence sources (and how to get the most from each)
Think of these as your four instruments. Rarely will you have all four; two or three that agree is often enough.
1. A lightweight or partial A/B test
Even when a fully powered test is off the table, a scrappy one often isn't. Run it on a single high-traffic segment, one geography, or for a short window.
- Why it helps: it's still randomized, so it retains a genuine (if noisy) causal signal — the strongest of the four.
- Its blind spot: underpowered, so results are wobbly and small effects may hide in the noise.
- How to strengthen it: pick your highest-traffic slice, commit to a single primary metric, and read it as "leaning positive/negative," not as a precise number.
2. Before-and-after (time logs / pre-post)
Measure the metric for a period before the change and the same-length period after.
- Why it helps: it's almost always available and intuitive to explain to stakeholders.
- Its blind spot: confounds. Seasonality, a concurrent marketing push, a holiday, or a competitor's move can masquerade as your effect. Before/after on its own is the easiest of the four to fool yourself with.
- How to strengthen it: use longer windows, compare year-over-year to cancel seasonality, write down any other changes that shipped in the window, and — if you can — watch a comparable segment that wasn't affected as a rough baseline. (That "what would've happened anyway" baseline is the whole idea behind incrementality, which is worth reading if this is a high-stakes call.)
3. Product analytics
Dig into funnels, retention curves, and behavioral flows to see whether the specific behavior you targeted actually moved.
- Why it helps: it's rich on mechanism — it shows you the exact step that changed, not just the top-line number.
- Its blind spot: it's correlational. It tells you what changed, not that your change caused it.
- How to strengthen it: zoom in on the precise step your change touched (a funnel analysis of just that transition), and segment down to only the users who actually experienced the change.
4. Qualitative research
Talk to users: interviews, targeted surveys, session replay, and support tickets.
- Why it helps: it's the only source that explains why, and it catches things the numbers can't — confusion, delight, a workaround you never anticipated.
- Its blind spot: small, self-selected samples that aren't statistically representative.
- How to strengthen it: target users who genuinely hit the change, and trust repeated themes over any single vivid quote.
How to triangulate, step by step
- State the hypothesis and the expected direction. "Simplifying the pricing page will increase checkout starts, because it reduces choice overload."
- Predict what each source should show if you're right — before you look. This pre-commitment is what stops you from rationalizing whatever you see. ("Before/after conversion up; the pricing→checkout funnel step up for new visitors; exit-survey mentions of confusion down.")
- Gather the independent signals. Pull each source separately so one doesn't bias how you read the next.
- Check for convergence. Do they point the same way? Agreement across independent, differently-flawed methods is your confidence.
- When they conflict, investigate — don't ignore. If before/after is up but the funnel and qual are flat, seasonality is the likely culprit, not your change. A conflict is a finding.
- Report a direction and a confidence level, and label the rigor. "We're fairly confident this helped (three of four signals agree); it's directional, not a precise lift." Honesty here is what makes the read trustworthy.
A worked example
Say you redesign a pricing page that every visitor sees — no way to A/B it. You triangulate:
- Before/after: checkout starts rose 9% in the four weeks after launch vs. the four weeks before — but you note a seasonal uptick, so you also check year-over-year and the lift holds at ~6%.
- Analytics: the pricing → checkout funnel step improved specifically for new visitors (who the redesign targeted), while returning visitors were flat — exactly the pattern you predicted.
- Qualitative: an exit survey shows fewer "the pricing was confusing" comments, and session replays show less hesitation on the page.
- Lightweight test: you can't A/B the page, but a quick new-vs-returning cohort comparison echoes the funnel finding.
No single line here is proof. Together, four differently-flawed signals all lean the same way — and the one that could have fooled you (before/after) was stress-tested against seasonality. That's a credible "yes, this worked," honestly labeled.
Why not just reach for fancy statistics?
There are advanced techniques — and a whole "rigor ladder" of quasi-experimental methods (geo experiments, difference-in-differences, synthetic control) — that can squeeze a more rigorous causal read out of messy situations. They have their place. But as Simon cautioned, unless there's a serious, serious need, he wouldn't reach for them first. They add real complexity, they often cost you statistical power on the metrics you actually care about, and — used without expertise — they can manufacture false confidence that's worse than an honest "we're fairly sure." Save them for high-stakes, expensive decisions where the extra rigor clearly pays for itself; for everything else, converging simple signals wins.
Common mistakes to avoid
- Treating before/after as proof. Without accounting for confounds, it's the easiest way to credit yourself with growth that was already happening.
- Cherry-picking the source that agrees with you. Pre-committing to predictions (step 2) is the antidote.
- Reporting false precision. If you have a direction, say "a direction" — don't dress it up as a 6.9% lift.
- Forcing heavy statistics to produce a "clean" answer the underlying data can't support.
- Skipping the qualitative leg. The numbers tell you what; only users tell you why — and the why is often the real insight.
Watch the full session
This was one exchange from a session full of hard-won lessons from thousands of experiments.
▶ Watch the full webinar recording
Find the gaps in how you measure impact
Most teams have a measurement blind spot they can't see from the inside. Start with our free Experimentation Gap Assessment for a quick read on where yours are — and if you want a deeper, hands-on diagnosis of how your team measures impact and runs tests, our Experimentation Readiness Audit maps exactly what to fix first.
Frequently asked questions
How do you measure impact without an A/B test?
You triangulate — combine several independent, imperfect signals (a lightweight or partial A/B test, before-and-after data, product analytics, and qualitative research) and check whether they converge on the same conclusion. No single source proves causation, but agreement across differently-flawed methods gives you a credible, defensible read.
What is triangulation in experimentation?
Triangulation is drawing on multiple independent sources of evidence to reach a conclusion no single source could support alone. Because each method has a different weakness, agreement between them makes it unlikely they're all wrong in the same direction — so you can be reasonably confident about the direction of impact even without a clean test.
Is before-and-after analysis reliable?
On its own, only weakly — it's vulnerable to confounds like seasonality, concurrent launches, and external events, which can look exactly like your effect. It becomes far more trustworthy when you control for seasonality (e.g. year-over-year), note other changes in the window, and cross-check it against analytics and qualitative signals.
When should you use advanced statistical methods instead?
Only when there's a serious need and the stakes justify the complexity — a high-cost, high-risk decision, for example. Techniques like difference-in-differences or synthetic control can add rigor, but they add complexity, can reduce power on your key metrics, and risk false confidence if used without expertise. For most decisions, triangulating simpler signals is the better call.
Can qualitative data prove a change worked?
Not by itself — samples are small and self-selected. But qualitative research is essential for explaining why a change did or didn't work, and it's a powerful corroborating signal alongside quantitative sources. Repeated themes across users carry more weight than any single quote.
