A/B test sample size: how many visitors you need and how long to run

A/B test sample size is the number of visitors each variant needs before you can trust the result. It comes from four inputs: your current conversion rate, the smallest lift worth detecting, the significance level and the statistical power. For a page converting at 3%, detecting a 20% relative lift at 95% confidence and 80% power takes about 13,900 visitors per variant. Divide that by your daily traffic and you know how long to run the test.

A/B test sample size is the number of users or sessions each variant must receive so that a test can reliably detect a chosen minimum effect, at a chosen significance level and statistical power, if that effect really exists.

What decides A/B test sample size?

Every sample-size calculation, including any ab test calculator you use, takes the same four inputs. Your traffic is not one of them. Traffic only decides how long it takes to collect the sample.

The baseline should come from the same page, audience and conversion event you plan to test, over a recent period of several weeks. If the test runs on mobile product pages, a sitewide average is the wrong number. If you are not sure which conversion event deserves to be the primary metric, settle that first; the guide to choosing marketing KPIs covers how to pick one metric that reflects real value.

What is minimum detectable effect, and is it relative or absolute?

The minimum detectable effect is the smallest true difference your test is designed to find. It is a business decision as much as a statistical one: ask yourself what lift would justify shipping the change, maintaining it and taking your team’s time.

MDE can be stated two ways, and mixing them up is the most common planning error:

• Relative MDE is a percentage of the baseline. A 20% relative lift on a 3% conversion rate means the variant must reach 3.6%.
• Absolute MDE is a difference in percentage points. A 0.6-point absolute lift on 3% also means 3.6%. A “20-point” absolute lift would mean 23%, which is a completely different test.

Calculators such as Evan Miller’s let you choose either mode, so always check which one is selected before you read the answer.

The MDE dominates the result because sample size scales with one over the effect squared. Halve the effect you want to detect and you need roughly four times as many visitors. This is why tests of small copy tweaks on low-traffic pages almost never finish: the effects are small and the required samples are enormous.

What do statistical significance and power actually mean?

You do not need the math to use these two settings well. Think of the test as a detector that can make two kinds of mistake.

Significance level (α) controls false alarms. At α = 5%, if the variant is truly no better than the control, there is a 5% chance the test still declares a winner. That is what “95% confidence” means in practice. It does not mean there is a 95% chance your variant is better.

Statistical power (1 − β) controls missed detections. At 80% power, if the variant really does improve conversion by at least your MDE, the test will flag it 80% of the time. One time in five, a real improvement will look like noise.

Ronny Kohavi and colleagues, whose work draws on large-scale experimentation programs at Microsoft and other large sites, describe a p-value threshold of 0.05 and 80% power as the industry standard when sizing a test (Kohavi, Deng and Vermeer, 2022). It is also the default in most calculators. You can move either dial, but each move has a price:

• Going from 80% to 90% power on the example below raises the sample from 13,911 to 18,623 visitors per variant, about 34% more.
• Loosening significance to 10% shrinks the sample but doubles your false-alarm rate. That can be acceptable for low-risk, easily reversed changes, as long as you decide it before the test.

How do you calculate A/B test sample size?

The standard approximation for comparing two conversion rates gives the sample size per variant:

n = (z_α/2 + z_β)² × [p1(1 − p1) + p2(1 − p2)] / (p2 − p1)²

p1      = baseline conversion rate (control)
p2      = p1 × (1 + relative MDE)
z_α/2   = 1.96   for 5% significance, two-sided
z_β     = 0.8416 for 80% power

Worked example (hypothetical numbers)

Say an online store’s product page converts 3% of visitors into purchases, and the team wants to detect a 20% relative lift.

1. Baseline: p1 = 0.03.
2. Target rate: p2 = 0.03 × 1.20 = 0.036. The absolute difference is 0.006, or 0.6 percentage points.
3. z-values: 1.96 + 0.8416 = 2.8016, and 2.8016² ≈ 7.849.
4. Variance terms: 0.03 × 0.97 = 0.0291 and 0.036 × 0.964 = 0.034704. Their sum is 0.063804.
5. Squared difference: 0.006² = 0.000036.
6. Put it together: 7.849 × 0.063804 ≈ 0.5008, and 0.5008 / 0.000036 ≈ 13,911.

The test needs about 13,911 visitors per variant, or 27,822 in total for a two-variant test.

The same paper gives a quick rule of thumb for 5% significance and 80% power: n ≈ 16 × σ² / δ², where σ² = p(1 − p) for a conversion rate and δ is the absolute difference. Here that is 16 × 0.0291 / 0.000036 ≈ 12,933. It lands slightly lower because it uses only the baseline variance, but it is close enough to sanity-check a calculator in your head.

How long should you run an A/B test?

Duration follows directly from the sample size:

days = (n per variant × number of variants) / daily visitors entering the test

If the product page above gets 2,000 eligible visitors a day, split evenly, the test needs 27,822 / 2,000 ≈ 13.9 days. Round up to 14 days, which is exactly two weeks.

Always round up to whole weeks, and run for at least one full week even when the math says less. Weekday and weekend visitors often behave differently, and a test that covers Monday to Thursday only compares a slice of your audience. Kohavi and colleagues also describe novelty and primacy effects: some users click around a new feature just because it is new, while experienced users may be slower at first with a changed layout, which favors the control. In their experience such effects rarely reverse an initial result, and many apparent trends turn out to be noise from wide early confidence intervals, but a run of full weeks gives them time to show and fade (Kohavi et al., 2012).

The table below keeps the same hypothetical page (3% baseline, 2,000 visitors a day) and changes only the MDE. It shows how quickly test length explodes as you chase smaller effects.

If the honest answer for your page is 30 weeks, the test is not worth running in that form. Skip ahead to the low-traffic options.

Why does peeking at A/B test results inflate false positives?

The 5% false-alarm rate holds only if you look at the result once, at the sample size you planned. Every extra look gives random noise another chance to cross the significance line. If you stop the moment it does, you will stop on noise far more often than 5% of the time.

Evan Miller’s essay “How Not To Run an A/B Test” makes this concrete. In his example, where the variant has no real effect and the experimenter checks after every new observation and stops at the first significant reading, the false-positive rate reaches 26.1%, more than five times the nominal 5%. Even ten scheduled peeks require a much stricter threshold, around 1% at each look, to keep the overall error near 5%.

Practical rules that prevent this:

• Fix the sample size and end date before launch, and write them down where the team can see them.
• Monitor for breakage, not for winners. It is fine to check daily that both variants load, that traffic is split as expected and that tracking fires. Just do not act on the conversion difference.
• If you must be able to stop early, use a method built for it. Sequential designs, such as the simple sequential test Evan Miller describes, set stopping boundaries in advance so that early stops stay statistically valid.

What can you do when your site has low traffic?

Low traffic does not rule out experiments, but it changes which ones are worth running. In rough order of impact:

1. Test bigger changes. A new offer, a redesigned checkout step or a different pricing presentation can plausibly move conversion by 30% or more, and the table above shows how much shorter that makes the test. Button-color tests belong to sites with very high traffic.
2. Measure an upstream metric with a higher baseline. With the same 20% relative MDE, a 1% baseline needs 42,691 visitors per variant, while a 10% baseline needs only 3,839. Add-to-cart or checkout-start often converts far more often than purchase. Keep purchase as a guardrail so you do not optimize clicks that never pay. Mapping your sales funnel stages shows which step has both a high rate and a real link to revenue.
3. Pool traffic across pages. Test a change on a page template, such as every product page, rather than one URL.
4. Run two variants, not five. Each extra variant needs its own full sample, and comparing many variants also raises the chance of a false winner.
5. Use a sequential design if you need the option to stop early, rather than peeking at a fixed-sample test.
6. Accept a looser setting deliberately. For cheap, reversible changes, 90% confidence may be a reasonable trade. Decide and document it before launch, never after seeing the data.
7. Do not force a test. If the required run is months long, user interviews, session reviews or a staged rollout with careful before-and-after reading may teach you more, faster.

How do you plan an A/B test with AdPix?

AdPix gives you the inputs for the calculation and a clean place to read behavior around the test.

• Find the baseline. The funnel and conversion path reports show how many visitors reach each step and where they drop off, so you can read the real conversion rate of the page and step you plan to test. Reports in AdPix are not sampled, so the counts you plan from are complete.
• Narrow the audience. On Starter and above, segments let you pull the baseline for exactly the audience you will test, such as mobile visitors or returning users, instead of a sitewide average.
• Estimate daily eligible traffic. Take the average daily visitors to that page over the last few full weeks, not a single good day.
• Size the test. Enter the baseline, MDE, significance and power into the free calculator below to get the sample size and duration.

If you are sizing a customer survey rather than a test, the survey sample size calculator answers that different question. And when the question is whether a marketing channel causes extra sales at all, rather than which page version converts better, that calls for a holdout experiment. Campaign lift measurement with holdouts is included from the Starter plan up, and true campaign impact (incrementality) measurement is part of Pro; the guide to attribution models explains why attribution alone cannot answer it.

Your next step: pick one page, pull its baseline for the last four full weeks, write down the smallest lift you would actually ship for, and run the numbers before anyone builds a variant.

A/B test calculator

Sample size and duration for your next experiment. The free plan covers 100,000 events a month, no credit card. Setup takes under five minutes.

Start freeTry it free