A/B Test Calculator
Is Variant B Actually Better, or Did It Just Get Lucky?
Two variants rarely produce identical conversion rates even when they perform identically in reality — random variation in who happens to visit and convert guarantees some difference every time. An A/B test calculator exists to answer the harder question underneath the raw numbers: is the observed gap large enough, given the sample sizes involved, to trust as a real effect rather than statistical noise?
The Formula
Rate B = Conversions B / Visitors B
Pooled Rate = (Conversions A + Conversions B) / (Visitors A + Visitors B)
Standard Error = √(Pooled Rate × (1 − Pooled Rate) × (1/Visitors A + 1/Visitors B))
Z-Score = (Rate B − Rate A) / Standard Error
This is a two-proportion z-test. The Z-score is converted into a p-value using the standard normal distribution, and the result is compared against the confidence level you set.
Worked Example
Consider a test where Variant A converts 520 of 10,000 visitors and Variant B converts 610 of 10,000 visitors:
| Quantity | Value |
|---|---|
| Rate A | 5.2% |
| Rate B | 6.1% |
| Pooled rate | 5.65% |
| Standard error | 0.00327 |
| Z-score | 2.7563 |
| Two-tailed p-value | 0.0058 |
| Confidence achieved | 99.42% |
At a 95% confidence threshold, this result is statistically significant — the achieved confidence of 99.42% clears the bar.
Standard Z-Score Thresholds
| Confidence Level | Critical Z-Score |
|---|---|
| 90% | 1.645 |
| 95% | 1.960 |
| 99% | 2.576 |
If the calculated |Z-Score| exceeds the critical value for your chosen confidence level, the result is statistically significant.
How to Use This Calculator
- Enter Variant A Visitors and Variant A Conversions.
- Enter Variant B Visitors and Variant B Conversions.
- Set your desired confidence level (95% is the common default).
- Select Calculate to get the z-score, p-value, achieved confidence, and whether the result is statistically significant.
Related Calculations
Once significance is confirmed, quantify the size of the win with the Conversion Lift Calculator.