How big should an A/B test sample be?
A valid test requires a sample size calculated before launch, based on the current conversion rate, the minimum improvement worth detecting, and the statistical confidence required. Not a test run until a result 'feels' significant. Checking results daily and stopping the moment one version pulls ahead is the single most common way commercial A/B tests produce false winners, because early leads in small samples regularly reverse as more data comes in.
What should you A/B test?
Testing a truly different hypothesis (a different offer, value proposition or page structure) produces far more learning than testing a button color or minor wording change. Cosmetic tests are popular because they're easy to build, not because they're likely to move the metric that matters.
Is an A/B test with no winner a failure?
A test that shows no significant difference isn't a failed test. It's a real result that rules out a hypothesis, which is useful information even though it produces no immediate lift. Teams that only count 'winning' tests as successful create pressure to run and report only safe, low-impact tests.
Why A/B Testing matters
A/B testing is the only reliable way to know whether a change actually caused an improvement instead of coinciding with one. Without it, teams attribute normal fluctuation or seasonal effects to changes that had no real effect, and repeat the same ineffective changes indefinitely.
Calling a winner too early
A test shows Variant B converting 15% better than Control after three days, and the team ships it immediately. Had the test run to its pre-calculated sample size, the gap would have narrowed to a statistically insignificant 2%. The early 'win' was normal sample noise, not a real effect, and the team shipped a change with no actual impact while believing it had found a meaningful improvement.
Common mistakes
Stopping the test at first apparent significance
Checking daily and stopping as soon as a result looks significant inflates the false-positive rate well beyond the 5% the test is supposed to guarantee. Calculate sample size upfront and commit to it.
Testing cosmetic changes over structural ones
Button colors and minor copy tweaks are tested far more often than offers, pricing presentation, or page structure, even though the latter reliably produces larger, more informative results.
Treating a null result as a failure
A test that finds no significant difference is a real, useful finding that rules out a hypothesis. It should be recorded and built on, not discarded as wasted effort.
Where we work on this