An A/B test shows two versions of a page, email or product to comparable groups of users and measures which one converts better. Because conversion is a yes/no outcome, the natural statistic is the difference between two proportions. Even if both versions perform identically, the observed rates will rarely match exactly, so the question is whether the gap is larger than random sampling would produce.
The two-proportion z-test answers that by pooling both groups to estimate the common conversion rate under the null hypothesis of no difference. The standard error built from that pooled rate turns the observed gap into a z-score; the p-value is the chance of a z at least that extreme if the versions really are equal. The confidence interval uses the unpooled standard error and shows the range of plausible true differences in percentage points.
Choose the confidence level before the test starts and keep the sample size fixed in advance. Checking the result every day and stopping as soon as it looks significant greatly increases the risk of declaring a winner that is not real.
How to use the A/B Test Calculator
- Visitors (sample size) in A
- Conversions in A
- Visitors (sample size) in B
- Conversions in B
- Confidence level
- Alternative hypothesis (B compared with A)
Everything is calculated in your browser, so results appear instantly and nothing you type is sent to a server.
How it works
Divide conversions by visitors to get each rate, pool them for the null standard error, and compute z = (p_B − p_A) ÷ SE. The p-value and critical z depend on the confidence level and on whether you test for any difference (two-tailed) or only for B beating A (right-tailed).
Worked example
Variant A converts 200 of 5,000 visitors (4.0%) and variant B 245 of 5,000 (4.9%). The pooled rate is 445 ÷ 10,000 = 4.45%, so the standard error under H₀ is √(0.0445 × 0.9555 × (1/5,000 + 1/5,000)) = 0.4124 percentage points.
z = 0.9 ÷ 0.4124 = 2.182. At 95% confidence the two-tailed critical value is ±1.96 and the p-value is 0.029, so the difference is significant. The relative uplift is 22.5% and the 95% confidence interval for the difference runs from 0.09 to 1.71 percentage points.
How to interpret the result
A significant result means the observed gap would be unlikely if both versions converted equally. It does not say the uplift is exactly 22.5%, as the confidence interval shows how small or large the true improvement might be. If the lower end of the interval is too small to matter commercially, consider running the test longer before rolling out the change.
Formula
p₁ = x₁ ÷ n₁ · p₂ = x₂ ÷ n₂ · p̂ = (x₁ + x₂) ÷ (n₁ + n₂) · z = (p₂ − p₁) ÷ √(p̂(1 − p̂)(1/n₁ + 1/n₂)) · CI = (p₂ − p₁) ± z* × √(p₁(1 − p₁)/n₁ + p₂(1 − p₂)/n₂)
Worked example
Inputs: Visitors (sample size) in A: 5,000; Conversions in A: 200; Visitors (sample size) in B: 5,000; Conversions in B: 245; Confidence level: 95%; Alternative hypothesis (B compared with A): Two-tailed (H₁: not equal).
Results: p-value = 0.03; z-score = 2.1823; Decision = Reject H₀ (no difference) at α = 0.05 – p = 0.0291 ≤ 0.05 and z = 2.1823 lies in the rejection region.; Critical z (from the confidence level) = ±1.96; Conversion rate A = 4 %; Conversion rate B = 4.9 %.
Key terms
- Conversion rate
- Share of visitors who complete the goal, conversions ÷ visitors.
- Pooled proportion
- Combined conversion rate of both groups, used for the standard error under H₀.
- Relative uplift
- (p_B ÷ p_A − 1) × 100%, the proportional improvement of B over A.
- Confidence level
- 1 − α; the long-run share of intervals that contain the true difference.
Frequently asked questions
How many visitors do I need for an A/B test?
It depends on the baseline rate and the smallest uplift you care about. Detecting a rise from 4% to 5% with 80% power at α = 0.05 needs about 6,750 visitors per variant, so use the power analysis calculator to plan.
Is a 95% confidence level always right?
It is a common default. Use 90% for low-risk tweaks you can easily revert, and 99% when a wrong decision is costly.
What is the difference between relative uplift and the percentage-point difference?
Going from 4.0% to 4.9% is a 0.9 percentage-point difference but a 22.5% relative uplift. Report both.
Can I stop the test when p drops below 0.05?
Not if you keep checking repeatedly. Peeking inflates the false-positive rate well above 5%; fix the sample size first or use a sequential testing method.
Should I use a one-tailed test?
Only if you decided beforehand that you would act only when B is better and would treat "B worse" the same as "no difference". Otherwise use two-tailed.
Why do the z-test and the confidence interval sometimes disagree near the cut-off?
The test uses the pooled standard error (assuming H₀) while the interval uses the unpooled one, so results right at the boundary can differ slightly.
When is the z-test not appropriate?
When conversions or non-conversions in a group are fewer than about 10; then use Fisher's exact test.
Sources
Guides on this topic
- How to Calculate Sample Size for a Survey (Formula + Examples): 385 responses give ±5% at 95% confidence for almost any large population. Here is the formula behind that number and how to adjust it for your survey.