.svg)
Most experimentation programmes hit the same ceiling. The team runs tests, documents results, and moves on, but the programme never builds on itself. Hypothesis generation takes weeks. Prioritisation is based on gut feel. Results get filed and forgotten. The bottleneck is not ideas or traffic, it is the manual overhead of running a rigorous programme at speed. This article breaks down the three reasons experimentation programmes stop compounding, and the AI-powered system that fixes all three without adding headcount.
There is a version of experimentation that most growth teams never reach, because the infrastructure required to run a compounding programme, one where every test makes the next one smarter is too manual to sustain at speed.
Most teams are stuck in a loop. They run an experiment. They document the result. They start the next sprint from scratch. Three weeks later, a new hypothesis gets written without referencing the last five experiments. ICE scores get filled in based on whoever is most opinionated in the room. Results sit in a Notion doc nobody reads.
That is not a compounding programme. That is a series of disconnected bets.
Here is what is actually breaking it and how to fix it.
Why Experimentation Programmes Stop Compounding
Three bottlenecks kill almost every programme at scale.
Hypothesis generation takes too long
Writing a strong hypothesis requires pulling together analytics data, session replay insights, customer feedback, and past experiment results, then synthesising them into a testable bet with a clear expected outcome and rationale.
Done manually, this takes days. Most teams do not have the bandwidth to do this thoroughly every sprint. So they default to gut feel dressed up as hypothesis. The result: hypotheses that are not grounded in evidence, tests that are unlikely to produce meaningful lift, and a programme that does not learn from itself.
Prioritisation is subjective
ICE scoring, Impact, Confidence, Ease, is the most widely used prioritisation framework in experimentation. It is also one of the most gamed. When scores are filled in manually, they reflect who is most senior in the room or which idea people are most excited about, not what the data actually suggests.
This is one of the analytics mistakes that kill experimentation programmes, subjective prioritisation that disconnects the experiment roadmap from the signals the business is actually generating.
Results do not feed back into the next round
Teams document results. They write up what they tested, what happened, and what decision they made. And then that documentation sits in a folder, disconnected from the next round of hypothesis generation.
The experiment that revealed returning users respond differently to a UI element than new users, does that insight automatically generate a new personalisation hypothesis? In most programmes, it does not. Someone has to remember it exists, find it, and apply it manually.
Without a system that automatically turns results into new hypotheses, every sprint starts from scratch. The programme runs experiments but never compounds.
The AI System That Fixes All Three
The fix is not more analysts or bigger research budgets. It is a connected system where AI handles the mechanical work: hypothesis generation, prioritisation scoring, result evaluation, so your team focuses on judgment and decisions.
The system connects three tools: Airtable as the central hypothesis and experiment repository, Claude as the AI reasoning layer, and Amplitude as the analytics data source. Five workflows run on top of this stack.
Workflow 1: Automated Hypothesis Generation
Every time an experiment concludes, the result: statistical outcome, segment-level observations, documented learnings, gets fed back into Claude. Claude reads the result in the context of your full experiment history and business model, and automatically generates new hypotheses based on what it finds.
Winners get iterated. Patterns across multiple experiments surface. The system learns with every experiment rather than resetting. Hypothesis generation speed improves by 10x compared to manual research.
Workflow 2: AI-Calibrated ICE Prioritisation
Instead of filling in ICE scores manually, the system scores every hypothesis automatically using your actual business data. Impact scores are calibrated to your conversion funnel and traffic volumes. Confidence scores are based on the strength of supporting evidence: session replay signals, analytics data, customer feedback, past experiment results.
The result is a prioritised experiment roadmap that reflects what your data says -- not what the loudest person in the room thinks. This is the 8-step A/B testing framework applied systematically rather than manually.
Workflow 3: Automated Result Evaluation
Evaluating an experiment result properly: checking statistical significance, confidence intervals, segment consistency, guardrail metrics, and practical significance, takes two to three hours per experiment when done manually. This is exactly where false winners happen when teams shortcut it under time pressure.
Claude connects to your analytics, pulls the relevant data, runs through the evaluation checklist automatically, and writes structured documentation of the result. Two to three hours becomes fifteen minutes. And because the evaluation is systematic rather than manual, the false winner rate drops significantly.
Workflow 4: Meeting Sync Capture
Stakeholder input from weekly syncs gets captured via Fireflies. Claude extracts decisions, observations, and new signals from the transcript, scores them against your hypothesis backlog, and adds them to Airtable automatically. Insights from client conversations flow directly into the experiment pipeline without anyone having to manually translate them.
Workflow 5: AI-Assisted Variant Design
Used selectively where it accelerates without compromising craft, Claude assists with variant creation alongside Figma. Not a replacement for design judgment, but a way to explore more directions faster before committing to a build.
What Changes When the System Runs
The outcomes are concrete. Hypothesis generation speed increases 10x compared to manual research. Result evaluation drops from two to three hours to fifteen minutes per experiment. Full experiment history: hypothesis, design, result, learning, is captured automatically rather than depending on someone remembering to document it. And the system gets leaner with every cycle as the AI has more context to work with.
This is what a compounding experimentation programme actually looks like. Not more experiments, smarter ones, built on everything that came before.
Who This Is For
This system is built for Heads of Growth and VPs of Product at D2C, SaaS, and Marketplace companies with 20,000 or more monthly sessions and solid event tracking already in place. Teams that are running experiments but feel like the programme is slow, manual, or not building on itself. Teams that want to scale experimentation velocity without scaling headcount.
If your programme is producing results but not compounding: this is the infrastructure gap.
See the full system live, free webinar on June 11
Zain is running a free 45-minute session on June 11 at 5:00 PM CET where he demos all five workflows live, in real Airtable, Claude, and Amplitude environments. You will see exactly how the system is built and how it runs end to end.
⏰ Thursday June 11, 2026 | 5:00 PM CET | Free
Want this system built for your programme?
If you are already running experiments and want to plug this infrastructure into your stack, book a free strategy session. We will review your current setup and show you exactly where AI can remove the manual bottlenecks.
FAQ
How do you scale an experimentation programme with AI?
Scaling experimentation with AI means automating the three biggest manual bottlenecks: hypothesis generation, ICE prioritisation, and result evaluation. Using a connected stack of Airtable, Claude, and Amplitude, every experiment result automatically generates new hypotheses, scores are calibrated to real business data, and results are evaluated and documented in fifteen minutes instead of two to three hours.
What is a compounding experimentation programme?
A compounding experimentation programme is one where every test makes the next one smarter. Results feed back into hypothesis generation automatically. Patterns surface across experiments. Prioritisation improves as the system accumulates more data. Most programmes do not compound because the documentation and feedback loop is manual -- insights get recorded but never automatically applied to the next round.
What tools do you need to scale experimentation with AI?
The core stack is Airtable for hypothesis and experiment management, Claude as the AI reasoning layer, and Amplitude for analytics and experiment data. Fireflies handles meeting sync capture. The five workflows that run on top of this stack cover hypothesis generation, prioritisation, result evaluation, meeting capture, and variant design assistance.
How long does it take to evaluate an A/B test result with AI?
With the automated result evaluation workflow, evaluating a complete experiment result -- statistical significance, confidence intervals, segment consistency, guardrail metrics, and documentation -- takes approximately fifteen minutes. The same process done manually typically takes two to three hours per experiment.
What is ICE scoring in experimentation and how does AI improve it?
ICE scoring rates experiment hypotheses on Impact, Confidence, and Ease to prioritise which tests to run first. Manually, these scores are subjective and often reflect opinion rather than data. AI-calibrated ICE scoring uses your actual business data -- funnel metrics, traffic volumes, historical experiment results, and supporting evidence strength -- to generate scores that reflect what the data actually says rather than what the team believes.
Do I need to replace my existing experimentation tools to use this system?
No. The system is built on top of your existing analytics and experimentation stack. Amplitude is used as the data source for result evaluation and hypothesis generation context. Your existing experimentation tool -- Statsig, Amplitude Experiment, GrowthBook -- continues to run experiments. The AI layer sits on top of the process, not inside the tools themselves.
