What is Sample Size in A/B Testing?
Sample size is the number of visitors (or sessions) that each variation in an A/B test must receive before you can draw reliable conclusions about which version performs better. It is calculated before the test begins and determines how long the test needs to run.
Running a test without reaching the required sample size is one of the most common mistakes in optimization. It leads to false positives (declaring a winner that is not actually better) and false negatives (missing a real improvement because the test was underpowered).
How to calculate Sample Size
Sample size depends on four inputs:
- Baseline conversion rate — Your current conversion rate before the test. Lower baseline rates require larger samples.
- Minimum detectable effect (MDE) — The smallest improvement you want the test to detect. Smaller effects require larger samples.
- Significance level (alpha) — Typically set at 5% (95% confidence). Lower alpha values (stricter standards) require larger samples.
- Statistical power (1 - beta) — Typically set at 80%. Higher power requirements mean larger samples.
Most teams use a sample size calculator rather than computing this manually. The standard formula is based on the normal approximation for comparing two proportions.
Practical example: If your baseline conversion rate is 3% and you want to detect a 15% relative lift (from 3.0% to 3.45%) with 95% confidence and 80% power, you need approximately 14,000 visitors per variation — or 28,000 total for a two-variation test.
Why it matters for eCommerce and SaaS
Undersized tests are the biggest source of wasted effort in CRO programs. When a team calls a test after 3 days because one variation is “winning,” they are likely seeing random noise, not a real effect. The result is that they ship changes that either have no impact or actively hurt performance — and they lose confidence in the testing process.
For eCommerce businesses, this often happens during sales events when someone wants to run a “quick test” on a promotional page. The traffic is there, but the test has not run long enough to account for day-of-week effects and visitor behavior variation.
For SaaS businesses with lower traffic, sample size planning is even more critical because it determines which tests are feasible at all. A pricing page that receives 2,000 visitors per month cannot support a test with a 5% MDE — the test would need to run for many months. Understanding this upfront prevents wasted planning time.
Common mistakes
- Peeking at results — Checking test results before reaching the required sample size inflates false positive rates. If you must monitor tests in progress, use sequential testing methods designed for early stopping.
- Ignoring the sample size calculation — Running a test “until it looks significant” is not a valid approach. Decide the sample size before the test starts and commit to it.
- Splitting traffic unevenly — An 80/20 traffic split requires a larger total sample size than a 50/50 split to achieve the same statistical power.
- Not accounting for multiple goals — If you are tracking 5 metrics, the probability of a false positive across any of them is much higher than 5%. Apply corrections for multiple comparisons.
How acceleroi approaches it
At acceleroi, sample size calculation is the first step in test design. We calculate the required sample size based on the client’s traffic, baseline metrics, and the MDE that represents a commercially meaningful improvement. We then translate that into a projected test duration so the client knows upfront how long the test will run. If the required duration exceeds 6 weeks, we either increase the MDE, choose a higher-traffic page, or recommend a different optimization approach. We never call tests early, and we never extend tests to “wait for significance” — both practices inflate error rates.
Related resources
- Get a free CRO audit to understand which tests are feasible at your traffic levels
- Read our blog for A/B test planning guides