Most PM candidates who reach an experimentation question know what an A/B test is. Fewer can explain why their experiment design is wrong, what sample size they need before launching, or what to do when results contradict their hypothesis but the product shipped anyway.

This article covers the full cycle: how to form a testable hypothesis, how to size an experiment, how to read results correctly, and the pitfalls that produce misleading conclusions. It also covers how to talk about all of this in an interview, where the question "walk me through how you'd design an experiment for this feature" is less about statistics and more about your decision-making process.

What a Good Hypothesis Looks Like

Every experiment starts with a hypothesis. Not a hope or a hunch, but a falsifiable statement that connects a specific change to a specific outcome.

A weak hypothesis: "We think the new onboarding flow will improve retention."

A strong hypothesis: "Replacing the five-step setup wizard with a single guided action on first launch will increase day-7 retention for new users by 3 percentage points, because users who complete at least one core action in session one have historically retained at 2x the rate of those who don't."

The strong version specifies who is affected, what the expected direction and magnitude of change is, and why you expect it. The "why" matters because it constrains your interpretation of the results. If the experiment succeeds but through a mechanism you didn't predict, that should make you cautious about shipping, not confident.

The three elements of a testable hypothesis:

A specific change to the product (the independent variable)
A primary metric that will move if your hypothesis is correct (the dependent variable)
A causal mechanism you believe connects the two

If you can't articulate the mechanism, you don't yet have a hypothesis. You have a guess.

Setting Your Primary Metric and Guard Rails

Before running an experiment, you need to decide what you're optimizing for and what you're protecting.

The primary metric is the single number that determines whether the experiment succeeds or fails. Choose one. Experiments with multiple primary metrics create ambiguity: what do you do when one goes up and the other goes down?

Guard rail metrics are metrics that must not move adversely while you're chasing the primary metric. If you're optimizing checkout conversion rate, your guard rails might be revenue per order and customer support ticket volume. A feature that boosts conversion by increasing the number of users who accidentally complete a purchase is not a win.

Common mistakes:

Using a proxy metric as the primary metric when you actually care about something downstream. Increasing click-through rate (CTR) is not the same as increasing revenue. Make sure the metric you're measuring is genuinely causally connected to the outcome you care about.
Ignoring guard rails until after launch. Define them before the experiment starts or you'll rationalize away every adverse signal.

Minimum Detectable Effect and Sample Size

The minimum detectable effect (MDE) is the smallest change in your primary metric that would be meaningful enough to act on. Setting the MDE before the experiment is one of the most commonly skipped steps, and skipping it produces experiments that are either underpowered (they can't detect a real effect) or overlong (they run longer than they need to).

How to set MDE:

Start with business significance, not statistics. Ask: "If this feature moved the metric by X%, would we ship it?" If a 0.5% improvement in checkout conversion doesn't justify the engineering cost to maintain the feature, your MDE is not 0.5%. If a 2% improvement is the threshold where the decision changes, MDE = 2%.

Once you have the MDE, use a sample size calculator (or the standard formula) to determine how many users you need in each variant. This depends on:

The current baseline rate of the metric
Your MDE
Statistical power (typically 80%): the probability of detecting a real effect when one exists
Significance level (typically 5%): the probability of concluding an effect exists when it doesn't

The practical implication: many features at mature companies can only generate a signal on metrics that are large enough to be detected in a two-to-four week window. If the traffic to your experiment surface is low, you either need a larger MDE (change your threshold for what's worth shipping), a longer experiment, or a different primary metric that's more sensitive.

In interviews, when you're asked to design an experiment, mention MDE explicitly. "The first thing I'd do is decide what change in [metric] would actually be meaningful to act on. That sets the MDE, which then determines how long this experiment needs to run before I can trust the results." That line separates candidates who know the mechanics from candidates who are guessing.

Statistical Significance and What It Actually Means

Statistical significance tells you how likely your results are, assuming the null hypothesis is true. A p-value of 0.05 means: if the variant had no real effect, there's a 5% chance you'd see a result this large by random chance.

It does not mean:

There is a 95% probability your hypothesis is correct
The effect you observed is the true effect size
You should ship the feature

Practical significance vs. statistical significance. A result can be statistically significant but too small to matter. At scale, with millions of users, you can detect a 0.01% improvement in CTR. That doesn't mean you should reorganize your product roadmap around it. Always ask: is this effect meaningful given the cost of shipping and maintaining this change?

Confidence intervals (CIs). Report CIs alongside p-values. A statistically significant result where the confidence interval runs from +0.2% to +3.8% is much less actionable than one where the CI is +1.5% to +2.5%. Wide CIs mean high uncertainty about the true effect size, even if you're confident an effect exists.

The Four Most Dangerous Pitfalls

1. Peeking

Peeking is checking your experiment results before the pre-determined end date and making a decision based on what you see. It dramatically inflates false-positive rates. If you check an experiment 20 times during its run and stop it when p crosses 0.05, you're not running at 5% false positive rate. You're running at something closer to 30%.

The discipline: set your end date before you start. Use sequential testing methods (like always-valid p-values or SPRT, the sequential probability ratio test) if you genuinely need the ability to stop early with valid results.

In interviews: "I'd pre-register the end date and not look at results during the run. If early stopping is a business requirement, I'd use a sequential testing framework that accounts for repeated looks."

2. Multiple comparisons

Every additional metric you test increases your chance of a false positive. If you test 20 metrics and use a 5% significance threshold for each, you should expect one false positive on average just by chance.

If you're testing multiple metrics, apply a correction. The Bonferroni correction is simple: divide your significance threshold by the number of comparisons (0.05 / 20 = 0.0025). More sophisticated methods like the Benjamini-Hochberg procedure control the false discovery rate (FDR) rather than the family-wise error rate, which is usually more appropriate for product experiments.

The simpler version for interviews: "I'd pre-specify one primary metric. Secondary metrics would be interpreted directionally, not used as decision criteria, precisely to avoid the multiple comparisons problem."

3. Sample ratio mismatch (SRM)

SRM happens when the actual split between your control and variant doesn't match the intended split. If you designed a 50/50 experiment and your variant ended up with 47% of traffic instead of 50%, something went wrong with randomization.

SRM almost always indicates an error in the experiment setup: a logging inconsistency, a bug in the assignment mechanism, or a data pipeline issue. Results from an experiment with SRM cannot be trusted. Check for SRM before interpreting any results.

4. Novelty effects

When users encounter a new feature, they engage with it out of curiosity, not because it's inherently better. A new notification format might see higher CTR in week one simply because it's different, not because it's more effective. That spike decays as users habituate.

If your experiment window is short and the feature is visible and new, your results may reflect novelty, not sustained value. Mitigations: run longer experiments for high-visibility surfaces, look for a second-week pattern (does CTR hold or decay?), and be skeptical of outsized early signals on features where the experience of "seeing it for the first time" is itself a factor.

Interpreting Results When They're Inconclusive

"No statistically significant result" is not the same as "the feature has no effect." It means you don't have enough evidence, given your sample size and run time, to conclude an effect exists.

Before concluding neutral, ask:

Was the experiment adequately powered? If you set MDE at 3% but the true effect is 1%, you wouldn't detect it.
Did you run long enough? If you stopped early, you may have missed a longer-term effect.
Was there segment heterogeneity? The global result might be neutral while a specific segment is strongly positive. Explore post-hoc, but treat those findings as hypotheses for a new experiment, not conclusions.

A neutral result from a well-powered, full-duration experiment is a genuine signal. It means either the feature doesn't move the metric, or it moves it by less than your MDE. Both are useful to know.

Talking About Experimentation in PM Interviews

Experimentation questions in interviews usually take one of three forms:

Design an experiment for a proposed feature. Walk through hypothesis, primary metric, guard rails, MDE, sample size, duration, and the criteria for shipping or killing. Don't skip MDE. It signals that you understand the connection between experiment design and decision-making, not just the statistical mechanics.

Here are results from an A/B test. What would you do? This is an interpretation question. Work through: Is the primary metric positive and significant? Are guard rails clean? Is the effect size practically significant? What do CIs say about the range of the true effect? Only after working through all of that say what you'd recommend. If results are ambiguous, say so explicitly and explain what additional evidence you'd want.

Tell me about an experiment you ran. Use this as an opportunity to show the full cycle. Choose an example where something was unexpected: the result went against your hypothesis, or you found an SRM, or you had to kill a feature that looked promising. Candidates who can talk about how they handled a surprising or adverse result demonstrate more expertise than those who only share clean successes.

The Connection Between Experiments and PRDs

Experimentation decisions belong in product requirements documents (PRDs) before a feature ships. A PRD that specifies "this will be A/B tested" without defining the hypothesis, primary metric, MDE, and duration is leaving the experiment design to chance.

For more on PRD structure, see How to Write a PRD. For how success metrics connect to the broader measurement framework, see How to Measure Product Success in PM Interviews. If an experiment returns a metric drop, the PM Metric Drop Root Cause Analysis guide covers how to diagnose what went wrong.

Cala Voice lets you practice experimentation questions out loud with an AI interviewer that challenges your hypothesis design, presses you on statistical reasoning, and gives feedback on how clearly you communicate under ambiguity.

Start your preparation →

A/B Testing for Product Managers: How to Design and Interpret Experiments