.svg)
Controlled Experiments Explained: How to Run One and Avoid Common Mistakes
The short version: A controlled experiment tests a single change by randomly splitting your audience into a control group (the current experience) and a treatment group (the change), while holding everything else constant — so any difference in the outcome can be credited to that one change. To run one well, state a clear hypothesis, randomize the split, calculate your sample size before launching, pick one primary success metric, and analyze only once the test reaches its planned size. The most common ways they go wrong: peeking at results early, having no predefined success metric, and using a sample too small to detect a real effect.
What a controlled experiment actually is
A controlled experiment isolates the impact of one change by comparing two groups that are identical except for that change: a control group that keeps the current experience, and a treatment group that gets the new version. Because users are assigned randomly and everything else stays the same, any measurable difference in results can be attributed to the change rather than to chance or outside factors. If you want the formal definition and how it fits alongside related terms, see our glossary entry on controlled experiments. The rest of this guide is about actually running one — and not fooling yourself in the process.
How to set up a controlled experiment, step by step
The mechanics are simpler than they sound. Five steps, in order, and the order matters. (If you want a deeper reusable process, our 8-step A/B testing framework expands on each of these.)
1. Start with a clear, testable hypothesis
A hypothesis is a prediction with a reason, not a hunch. "Let's try a green button" isn't testable; "Changing the CTA copy from Start free trial to Get started free will increase trial signups, because it lowers perceived commitment" is. State the change, the expected direction, the metric it should move, and why. That reasoning is what makes the result worth learning from either way.
2. Split into control and treatment — randomly
Randomly assign each user to one group or the other. Randomization is the whole point: it's what makes the two groups comparable, so the change is the only systematic difference between them. If you split by anything non-random (say, new users in one group and returning users in the other), you've baked in bias before you've started.
3. Calculate your sample size before you launch
Decide three things up front: your baseline rate for the metric, the smallest effect worth detecting (your minimum detectable effect), and your confidence and power levels. Those determine how many users you need per group — and roughly how long the test must run. Working this out beforehand is what stops you from ending a test the moment it looks good. Our guide to A/B testing statistics — p-values, confidence intervals, and sample size walks through the math.
4. Pick one primary success metric
Name the single metric that decides win or lose before the test starts. Track secondary and guardrail metrics too (you don't want to lift signups while quietly tanking retention), but one metric is the referee. Multiple "primary" metrics means you'll find something that moved and call it a win.
5. Run to completion, then analyze
Let the experiment run until it hits the sample size and duration you planned, ideally covering full business cycles so you're not misled by weekday-vs-weekend behavior. Only then check the result: is the difference statistically significant, and is it practically significant — big enough to matter? A tiny, "significant" lift on a huge sample can still be commercially irrelevant.
Common mistakes that quietly ruin the result
Most failed experiments aren't wrong because the idea was bad. They're wrong because of process.
- Peeking early and stopping when it looks significant. Checking repeatedly and calling it the moment you see p < 0.05 dramatically inflates false positives — you'll declare winners that don't hold up. Set your stopping rule in advance and stick to it. This is the single most common way teams ship false winners.
- No clear success metric. If you didn't define the primary metric first, you'll rationalize whatever number looks best after the fact. That's not measurement, it's storytelling.
- A sample size that's too small. Underpowered tests either miss real effects or produce noisy "wins" that vanish when you try to replicate them. If you can't reach a reasonable sample in a reasonable time, that's a signal in itself.
- Changing more than one thing at once. If treatment differs from control in three ways, a win tells you something worked — but not what. Isolate the variable.
When to use a controlled experiment (and when not to)
Reach for a controlled experiment when you can randomly split your audience and you need real causal certainty about a specific change — a new onboarding flow, pricing page, or feature. It's the strongest evidence you can get that a change caused an outcome.
It's the wrong tool when you can't randomize or don't have the traffic. A change everyone sees at once (a company-wide pricing update), a low-traffic B2B product where a test would take months, or a broad strategic bet all call for other methods — before/after analysis, cohort comparisons, or qualitative research — with the caveat that those give weaker causal claims. And if you're not sure your product data and tooling are even set up to run trustworthy experiments in the first place, an experimentation readiness audit will tell you what to fix before you waste a quarter on tests you can't trust.
Get controlled experiments right from the start
Running one experiment well is a process. Building a team that ships trustworthy experiments consistently is a system. If you want help standing that up — from instrumentation to analysis to a repeatable cadence — our experimentation programs are built for exactly that.
Frequently asked questions
What's the difference between a controlled experiment and an A/B test?
An A/B test is the most common type of controlled experiment — two variants (A and B), users split randomly, one change tested. "Controlled experiment" is the broader term and can include more than two variants or more complex designs, but the core principle is identical: hold everything constant except the change you're measuring.
How long should a controlled experiment run?
Long enough to reach the sample size you calculated up front, and long enough to cover full business cycles — usually at least one to two complete weeks so weekday and weekend behavior are both represented. Don't stop early just because the result looks significant.
What sample size do I need?
It depends on your baseline conversion rate, the smallest effect you want to detect, and your chosen confidence and power. Smaller expected effects need much larger samples. Calculate it before launching rather than checking as you go.
Why is peeking at results a problem?
Every time you check and consider stopping, you give the experiment another chance to cross the significance line by luck. Repeated peeking inflates your false-positive rate, so you end up "winning" with changes that have no real effect. Decide your stopping point in advance.
When shouldn't I run a controlled experiment?
When you can't randomly assign users, when traffic is too low to reach significance in a sensible timeframe, or when the change is all-or-nothing for everyone. In those cases, lean on before/after analysis, cohort comparisons, or qualitative research — and accept that the causal evidence is weaker.
