.svg)
📺 What is Statsig? (And Who Is It For?) Beginner's Guide
In the video above, Zain — Adasight's experimentation lead — gives a beginner's tour of Statsig. This article picks up where our feature management guide left off and digs into the part Statsig is best known for: experimentation.
The short version: Statsig's experimentation tools let you run controlled tests, read results against primary and secondary metrics, and plan tests properly before you launch. Its standout features are power analysis (which tells you the traffic and time you need to reach a statistically significant result), layers and holdouts (which let you run many experiments at once without them interfering), and multi-armed bandit tests. Because experiments run on the same feature gates used for rollouts, the tool that controls what's live is also the one that measures what works.
What experimentation in Statsig covers
At its core, Statsig is built for experimentation. Beyond a simple A/B test, it gives you the tools to run many experiments at once, plan them so they'll actually produce a clear answer, and read results against the metrics you care about. It's a big reason Statsig shows up in our roundup of the best A/B testing tools in 2026. This post walks through that experimentation side specifically — the natural next step after feature management, since the two share the same underlying plumbing.
Running an experiment and reading results
When you set up an experiment in Statsig, results stream in against the metrics you've chosen: a primary metric that decides the outcome, plus secondary metrics for guardrails and context. You can run "pulse checks" to see how an experiment is trending mid-flight, and choose different experimentation methods depending on what you're testing. The key discipline — true in any tool — is to name your primary metric before you start, so the result can't be cherry-picked after the fact.
Power analysis: plan before you run
The feature that wins a lot of teams over is power analysis. Before you launch, you pick a primary metric and run a power analysis to see how much traffic and how long you'll need to reach a statistically significant result. It also helps you set your minimum detectable effect (MDE) — the smallest change worth catching — and the statistical power you want.
In plain terms, it tells you up front whether your experiment can even succeed given your traffic, so you don't burn three weeks on a test that was never going to reach significance. If terms like statistical significance and MDE feel fuzzy, our guide to A/B testing statistics breaks them down.
Layers, holdouts, and exclusion groups
Once you're running more than one experiment, tests can start to interfere with each other. Statsig manages that with a few concepts:
- Layers let you run multiple parallel experiments — for different parts of your app or site — while keeping them cleanly separated. Users assigned to one experiment sit in a layer, and layers can have defined relationships with one another.
- Exclusion groups stop the same users from landing in conflicting experiments at the same time.
- Holdouts deliberately keep a slice of traffic — say 5% — out of all your feature gates and experiments, so you can measure the combined impact of everything you've shipped against a clean, untouched baseline.
That last one is quietly powerful: it's how you answer "has all our experimentation actually moved the needle?" rather than just "did this one test win?"
Multi-armed bandit tests
Statsig also supports multi-armed bandit experiments — a method that shifts traffic toward the better-performing variant while the test runs, instead of splitting evenly and waiting until the end. It's useful when you'd rather maximize results during the test than get the cleanest possible academic read.
How this connects back to feature gates
None of this is separate from feature management — it's built on it. The same gate that rolls a feature out to 5% of users is what assigns users to a control or treatment group in an experiment. That's why Statsig's feature management and its experimentation tools are really two sides of one system: one controls what's live, the other measures whether it worked.
Fuel your experiments with better ideas
Statsig gives you the machinery to run experiments well — but you still need a steady stream of ideas worth testing. Our free Hypothesis Bank Playbook shows how to generate and score experiment ideas so your testing pipeline never runs dry.
Build a real experimentation practice
Running the occasional experiment is easy; building a team that ships trustworthy experiments consistently is the hard part — the right metrics, enough power, and a repeatable cadence. That's exactly what our Experimentation Programs are built to set up.
Frequently asked questions
What is power analysis in Statsig?
Power analysis is a pre-launch calculation that tells you how much traffic and how long an experiment needs to reach a statistically significant result. In Statsig, you pick a primary metric and run it before starting, so you know upfront whether the test is viable given your traffic — and can set your minimum detectable effect and statistical power accordingly.
What are layers in Statsig?
Layers let you run multiple experiments in parallel without them interfering. Users in one experiment are grouped into a layer, and layers keep separate tests cleanly isolated — useful when you're experimenting across different parts of your product at once.
What is a holdout in Statsig?
A holdout keeps a portion of your traffic (for example, 5%) out of all your feature gates and experiments. That untouched group acts as a baseline, letting you measure the cumulative impact of everything you've shipped, not just the effect of a single test.
What's the difference between a feature gate and an experiment in Statsig?
A feature gate controls whether a feature is on and who sees it; an experiment uses that same gating to split users into control and treatment groups and measure the difference between them. The gate is the mechanism; the experiment is the measurement built on top of it.
What is a multi-armed bandit test?
It's an experiment that automatically shifts more traffic to the better-performing variant as results come in, rather than holding a fixed split until the end. It trades some statistical cleanliness for better outcomes during the test itself.
.png)
