.svg)
Why Experimentation Programs Fail to Scale (and What Actually Works)
At our recent webinar, Adasight x Cherto: 5 Lessons Learned from 1,000s of Experiments, two comments landed on the same nerve. Adasight's Gregor made an aside about the "feature factory": now that AI has made almost everyone an engineer, the world is suddenly realizing what the experimentation community has known forever — that you have to measure outcomes, not output. And Dr. Simon Jackson, explaining why he's building his own company, noted that even with the cutting-edge in-house tools and on-market platforms he's worked with, companies still struggle to scale experimentation.
Prefer to watch it live?
Put those together and you have today's question: why do experimentation programs stall as they grow — and what actually makes them scale?
The short version: Experimentation programs fail to scale when teams optimize for output — how much they ship — instead of outcomes — whether what they shipped actually moved a metric. AI makes this worse: now that almost anyone can build and ship quickly, output explodes while learning stays flat, widening the gap between activity and impact. Scaling a program isn't about running more tests or buying a better tool. It's about building the organizational discipline to measure outcomes, keep results trustworthy as volume grows, and let anyone test — so that more shipping actually becomes more learning.
First, what "scaling experimentation" actually means
Most teams hear "scale experimentation" and picture one of two things: run more tests, or buy a better platform. Both miss it.
Scaling a program means producing more trustworthy decisions per unit of time, across more of the organization. The unit of progress isn't a shipped feature or even a completed test — it's a validated learning. A team running 200 tests a year that can't trust half of them, or that ships regardless of the result, hasn't scaled experimentation. It's scaled activity. Those are not the same thing, and conflating them is the root of most of the failures below.
Why programs fail to scale
1. The feature factory: mistaking output for outcomes
The most common failure is cultural. Teams measure themselves by what they ship — roadmap items closed, features launched, velocity — because output is visible, countable, and satisfying. But output only feels like progress. A feature that ships and changes no metric isn't progress; it's cost. When shipping is the scoreboard, an experimentation program becomes a rubber stamp: tests get run, but the decision to ship was never really on the table. This is exactly the gap we've written about in why teams ship 10× faster but learn at the same speed — motion mistaken for movement.
2. AI just poured fuel on the fire
Here's Gregor's point, sharpened. AI has made it dramatically easier to build and ship — anyone can spin up a feature now. That's genuinely powerful, but it has a dark side: if your bottleneck was output, AI removes it, and output explodes. Meanwhile your ability to learn — to figure out whether all that shipping helped — stays exactly where it was. The gap between how much you ship and how much you understand widens fast.
As Gregor put it, we can almost thank AI for making the problem obvious: an AI-accelerated feature factory just ships the wrong things faster. The teams that win with AI point it at learning, not just building — a distinction we get into in how to scale experimentation with AI.
3. Trustworthiness breaks under volume
Even teams that genuinely want to learn hit a statistical wall as they scale. More tests mean more false positives, more chances for metrics to drift by luck, and more cannibalization between teams — unless the discipline scales alongside the test count. Without shared primary metrics, guardrails, and stopping rules, a high-volume program produces a stream of confident-looking "wins" that quietly don't replicate. Volume without trust isn't scale; it's noise with a bigger budget. (Several of the analytics mistakes that kill experimentation show up precisely at this stage.)
4. It stays a bottlenecked specialty
In many companies, experimentation is owned by a small central team. Everyone else submits requests and waits in a queue, or has their idea overruled by the highest-paid person in the room. That structure can support a few good tests — it cannot support a program. If every experiment has to pass through one team's backlog, your throughput is capped no matter how good your tooling is.
5. Teams expect the tool to be the program
This is Simon's own hard-won finding: even the best tools — cutting-edge in-house builds and mature on-market platforms alike — fall short at genuinely facilitating experimentation at scale. A platform is necessary but nowhere near sufficient. Buying a tool and declaring an experimentation program is like buying a treadmill and declaring yourself fit. The program is the process, the culture, and the capability wrapped around the tool — and that's the part that's actually hard to build.
What actually scales a program
Anchor everything on outcomes
Define the metric that matters and hold shipped work accountable to it. The scoreboard question shifts from "did we ship it?" to "did it move the number?" This single reframe kills the feature factory, because it makes output without outcome visibly worthless.
Build trust before you chase volume
Scale your discipline in lockstep with your test count: an agreed primary metric per test, guardrails that catch cannibalization, and stopping rules that prevent false wins. A trustworthy program running 50 tests beats an untrustworthy one running 500 — because the first one's decisions are real.
Democratize, and retire the HiPPO
Take a page from the companies that experiment at massive scale: let anyone run a test, and replace opinion-based vetoes with evidence. The catch that makes this safe is the discipline above — with shared guardrails in place, you can decentralize who tests without losing control of quality. Democratization without guardrails is chaos; guardrails without democratization is a bottleneck. You need both.
Point AI at learning, not just output
AI is only a gift to a program that aims it correctly. Used to generate and prioritize hypotheses, speed up analysis, and automate the tedious parts of running trustworthy tests, AI raises your learning velocity. Used only to ship more, it accelerates the feature factory. Same tool, opposite outcomes — the difference is entirely in what you point it at.
Treat it as a capability, not a purchase
The durable version of scaling is organizational muscle: training so teams can design their own tests, clear ownership, and a repeatable process. That's the part no subscription includes, and the part that keeps working after the novelty wears off.
The reframe that ties it together
More shipping was never the goal — more learning was. That's the thing the experimentation community understood long before AI, and the thing every team is now being forced to confront as output gets cheap. Scaling an experimentation program isn't about doing more; it's about making sure that what you do more of is learning. Get that right and volume compounds in your favor. Get it wrong and you've just built a faster way to be wrong.
Watch the full session
This came out of a session packed with lessons from thousands of experiments.
▶ Watch the full webinar recording
See whether your program is built to scale
Most teams can't tell from the inside whether they're scaling learning or just output. Start with our free Experimentation Gap Assessment for a quick read on where your program is leaking — and if you're ready to build the process, culture, and capability that actually scale, our Experimentation Programs are built for exactly that.
Frequently asked questions
Why do experimentation programs fail to scale?
Usually because teams scale output (how much they ship) without scaling outcomes (whether it worked) or the discipline that keeps results trustworthy. Add bottlenecked ownership and an over-reliance on tooling, and the program produces more activity but not more reliable learning.
What is a "feature factory"?
A feature factory is a team or company that measures success by how many features it ships rather than by the impact those features have. It optimizes for output, treats shipping as progress, and rarely checks whether what it built actually moved a meaningful metric.
What's the difference between output and outcomes?
Output is what you produce — features shipped, tests run, roadmap items closed. Outcomes are the results those things cause — a metric that moves, a user problem solved. Mature experimentation programs hold output accountable to outcomes; struggling ones celebrate output on its own.
Does AI help or hurt experimentation?
Both, depending on where you point it. Aimed at output, AI accelerates the feature factory — you ship more, including more of the wrong things. Aimed at learning (generating and prioritizing hypotheses, speeding analysis, automating trustworthy test setup), it raises how fast you learn, which is the thing that actually scales a program.
How do you scale an experimentation program?
Anchor on outcomes, build trustworthiness (shared primary metrics, guardrails, stopping rules) before chasing volume, let anyone run tests while keeping guardrails in place, and treat experimentation as an organizational capability rather than a tool you bought. The goal is more trustworthy decisions per unit of time — not more tests for their own sake.
