Notes · A/B Testing · By olivia bedford
Do you check your data while your A/B test is running?
Went down a bit of a peeking rabbit hole these past couple weeks — on why we essentially open ourselves up to an increased probability of implementing a false positive the more we check the data mid-test.
“Data monitoring,” or peeking, seems like the responsible thing to do, right? Looking at the data and keeping track of it like a proper scientist.
Well, statistically… apparently no. Peeking actually compounds the probability that we implement a false positive. It mathematically breaks the test.
The 5% you think you’re buying
When you run an A/B test at 95% confidence, you’re accepting a 5% false positive rate. So if you ran 100 tests where the variant had zero real effect, you’d wrongly declare a winner about 5 times out of those 100.
But — and this is the part that gets glossed over — that 5% only holds if you look at the results once. At the end. At a sample size you decided on before the test began.
That’s the deal you’re actually striking with a 95% confidence level. Not “5% risk, however I choose to check it.” Just: one look, one pre-committed sample size, one decision.
Same test, but you peek 10 times
Now run the same scenario, but you check in on the test 10 times over its course instead of once at the end.
At each peek, you’re asking “is this significant yet?” — and you’re making a decision: is this significant enough to stop? That decision is a whole separate test. You’re not running one test anymore. You’re running ten, every time you peek.
So if each check carries a 5% chance of a false positive, what’s the probability of getting at least one false alarm across all ten?
The maths:
Probability of not getting a false positive on any single check: 0.95
Probability of never getting one across 10 checks: 0.95^10 = 0.5987
Probability of getting at least one false positive: 1 − 0.5987 = 0.4013
Your real false positive rate after 10 peeks is ~40%, not 5% anymore.
And it keeps climbing the more you look:
Number of peeks
P(at least one false positive)
1
5%
5
~23%
10
~40%
20
~64%
50
~92%
(This treats each peek as independent, which isn’t exactly true — consecutive checks share overlapping data, so they’re correlated, and the real number is a bit lower than this simple formula suggests. But the direction and the scale of the problem hold: more looks, more risk, and it climbs fast. Statisticians call this “alpha inflation” from repeated significance testing.)
Push it further and the picture gets worse, not better. If you’re willing to keep monitoring indefinitely and stop the moment you cross your significance threshold, you will eventually cross it — even if there is genuinely no effect at all. Given enough looks, hitting “significant” isn’t a matter of if, just when. The test statistic wanders like a random walk, and a random walk with no drift will cross almost any fixed line eventually, given infinite time to wander.
Why does this happen?
Every time you peek and ask “is it significant?”, you’re spending a slice of your 5% error budget. The 95% confidence threshold was calibrated assuming you spend it exactly once. Peek ten times, and you’ve spent it ten times over — the test doesn’t know or care that you’re “just checking.” Statistically, each check is a fresh roll of the dice, and the probability of at least one of those rolls coming up a false alarm only grows with every roll you take.
This is sometimes called the “optional stopping” problem: the moment your stopping rule depends on the data itself (“stop when p < 0.05,” rather than “stop at n = 10,000 regardless”), the guarantees of your significance threshold no longer apply as advertised.
So why 5% in the first place?
Worth remembering: 5% isn’t a law of nature, it’s a convention. It traces back to Ronald Fisher in the 1920s, who proposed it as a reasonable round number for “surprising enough to take seriously.” It represents a tradeoff — set the threshold lower and you catch fewer false positives but also miss more real, smaller effects; set it higher and the reverse. 5% became a shared standard because it’s a workable balance, and because having everyone agree on one number lets results be compared across experiments and teams.
That’s exactly why peeking is so corrosive — it quietly erodes a number that the whole discipline of significance testing depends on being fixed and agreed-upon.
What to actually do instead
None of this means you can’t look at your dashboard. It means the decision to stop needs a method built for repeated looks:
Fix your sample size or test duration up front, based on a power calculation, and don’t make stop/go decisions until you hit it. Looking for curiosity is fine — deciding based on what you see mid-test is where it breaks.
Use sequential testing methods designed for this exact problem — group sequential designs (O’Brien-Fleming, Pocock boundaries) or alpha-spending functions divide your error budget across a fixed number of planned interim looks, so your cumulative false positive rate stays at 5% even with multiple checks.
Use always-valid inference — methods built into platforms like Optimizely and Statsig let you monitor continuously without inflating your false positive rate, because the math accounts for continuous peeking from the start.
Consider a Bayesian approach, which frames the question differently (probability the variant is better, given the data so far) and sidesteps the classical stopping-rule problem, though it brings its own assumptions to interrogate.
The dashboard updates whether you look at it or not. The fix isn’t willpower — it’s picking a method whose math already accounts for how many times you’re actually going to check.
I find it fascinating that psychology can break maths.
Book a call
all rights reserved @misoagency2026