.svg)
How to Build a Hypothesis Backlog: Generate and Score Experiment Ideas
Most experimentation programs don't stall because the team can't run tests. They stall because they run out of good ideas to run. The first sprint is easy — everyone has opinions. By the third, the well is dry, prioritization has turned into a debate, and the program quietly loses momentum. The fix isn't more brainstorming; it's a system — a hypothesis backlog that stays full on its own.
The short version: Most experimentation programs stall not because teams can't run tests, but because they run out of good ideas to test. A hypothesis backlog fixes that — a single source of truth where every idea is captured in a consistent format, scored for priority, and continuously refilled, both proactively (from analytics, session replays, and user feedback) and reactively (from the results of finished tests). The standard way to rank ideas is ICE — Impact, Confidence, Ease — and the key discipline is scoring each idea before you debate it, so prioritization is driven by evidence, not by whoever argues loudest.
Why programs run dry: three bottlenecks
Before the fix, name the failure. Three bottlenecks kill most programs before they ever compound:
- Generation is slow and manual. Digging through session replays, analytics, UX audits, and customer reviews for ideas takes weeks. Most teams surface three to five and run out of steam.
- Prioritization is subjective. Scoring ideas in a room ends up biased toward whoever speaks loudest, with no objective check on whether the score reflects real business impact.
- Results don't feed back. When a test finishes, the learning ends up in a slide deck instead of becoming the next hypothesis. Every sprint restarts from zero rather than building on the last.
A hypothesis backlog is the structure that solves all three at once.
Step 1: Write hypotheses in a consistent format
A backlog is only useful if every idea is captured the same way — enough structure that anyone (or any tool) can score it later. A reliable format:
"We believe that [change] for [audience] will cause [primary metric] to [increase/decrease] because [evidence source + observation]."
For example: "We believe that adding security trust badges next to the pay button for all checkout users will increase payment success rate, because session replays show hesitation at the payment form and reviews cite security concerns as a reason for drop-off."
Notice what that format forces: a specific change, a specific audience, one primary metric, and — crucially — the evidence behind the idea. A hypothesis without evidence is just an opinion with better grammar.
Step 2: Generate ideas continuously, from two directions
The goal is a self-refilling backlog, which means pulling ideas from two arms:
Proactive — mining your existing signals for opportunities: funnel drop-offs and analytics, session replays and rage-click data, UX audit findings, customer reviews, and your business KPIs. This is what fills the backlog at the start of a program (a strong start can surface 30+ hypotheses in the first couple of weeks). If you want to accelerate this, AI can do a lot of the heavy lifting — we cover that in how to generate A/B test hypotheses automatically using AI.
Reactive — the compounding arm. Every time a test finishes, its result becomes the seed for new ideas: what won, why, what the segments showed. A winning test spawns follow-up variations; a losing one reframes the assumption. This is how each sprint makes the next one smarter instead of starting from scratch.
Step 3: Score with ICE — in the right order
Once ideas are in the backlog, you need to rank them. The most widely used framework is ICE:
- Impact — how much will this move the metric if it works?
- Confidence — how strong is the evidence behind it?
- Ease — how much dev effort and time will it take?
But the framework matters less than the sequence. The discipline that removes bias is this: have your team score each hypothesis independently, before any discussion — and before an AI or data model scores it. If the data-driven score lands first, everyone unconsciously anchors to it. Two independent scores let you see where human judgment and the evidence disagree — and that gap is the conversation worth having, not a problem to smooth over.
A real example of why the gap matters: a team once scored a trust-badge idea low on Confidence because they'd previously tested review widgets and seen nothing move. That context lived in no database — it surfaced in a weekly sync. Capturing it changed how every future idea of that type got weighted. The disagreement was the insight. (For more ways to structure prioritization, see our guide to prioritising your hypotheses.)
Step 4: Close the loop
This is what turns a backlog into a compounding engine rather than a to-do list. When a result lands, it does three things: it gets documented (decision, key learning, revenue impact, segment findings), it feeds the reactive arm to generate child hypotheses, and it grows a parent/child tree of ideas that gets richer with every test. Do this consistently and the bottleneck flips — from "we don't have enough ideas" to "we don't have enough traffic to run them all." That's the good problem to have.
It also matters because the odds demand volume. Across programs, a typical A/B test win rate sits around 12% — most ideas won't win, so more quality shots on goal is the only reliable route to more wins. A backlog is how you keep taking them.
What you need before this works
A hypothesis backlog rewards clean foundations. Before you build one, make sure you have:
- Trustworthy analytics. Ideas generated from misfiring events or incomplete funnels are built on bad signals. Fix tracking first.
- Enough traffic to test. As a rough guide, ~20,000+ monthly sessions on the page you're testing to reach significance in a reasonable runtime.
- Agreed primary metrics. You can't score Impact if the team hasn't defined what success looks like. Lock your primary and guardrail metrics up front.
- A weekly cadence. Generation can be automated, but a human still makes the final call. A 30-minute weekly review of the scoring gaps is enough to keep the backlog prioritized.
Get the exact system — Claude + Airtable
Everything above is the method. If you want the implementation — the Airtable structure, the generation prompts, and the ICE scoring workflow wired end to end with Claude — we've packaged it into a free, step-by-step playbook.
👉Download The Hypothesis Bank Playbook →
It's one workflow in a larger system. If you want to see how a full backlog connects to AI-powered prioritization, result evaluation, and variant design, our Experimentation Growth Engine turns the whole loop into a compounding growth engine — and our broader guide to scaling experimentation with AI shows where this fits among the five workflows.
Frequently asked questions
What is a hypothesis backlog?
It's a single, structured source of truth for every experiment idea your team has — each captured in a consistent format, scored for priority, and continuously refilled from analytics, user research, and the results of past tests. It exists so a program never runs out of ranked, ready-to-test ideas.
How do you write a good experiment hypothesis?
Use a consistent format: "We believe that [change] for [audience] will cause [primary metric] to [increase/decrease] because [evidence]." The evidence clause is what separates a real hypothesis from an opinion — every idea should be backed by a signal from analytics, session replays, or customer feedback.
What is ICE scoring?
ICE ranks ideas on three dimensions — Impact (how much it'll move the metric), Confidence (how strong the evidence is), and Ease (how much effort it takes). The trick is to score each hypothesis independently before discussing it, so prioritization reflects evidence rather than the loudest voice in the room.
How do you keep an experimentation backlog full?
Generate ideas from two directions: proactively, by mining analytics, session replays, UX audits, and reviews; and reactively, by turning every finished test's result into new follow-up hypotheses. The reactive loop is what makes the backlog self-sustaining instead of something you refill manually each sprint.
How many test ideas should a team have at once?
Enough that traffic — not ideas — is your constraint. Mature programs often hold 30+ active hypotheses with several variations each. Once you're there, the question shifts from "what should we test?" to "which of these do we run first?", which is exactly where you want to be.
