z-test Calculator

Why this is usually the wrong test for a mean

Worth being direct on a page called “z-test Calculator”: if you are testing a mean, you probably want a t-test.

The z-test assumes you know the population standard deviation σ. Not an estimate from your data — the actual value, known before you collected anything.

Almost nobody has that. You compute s from the same sample that produced the mean, and s carries its own uncertainty. Student’s t exists precisely to account for it, with heavier tails that widen as the sample shrinks.

Using z with an estimated σ makes the test anti-conservative: p-values come out too small and rejections too frequent, because the extra uncertainty has been ignored. The tool shows the size of that on your own numbers by computing both.

The old rule — “use z when n > 30” — is a convenience from the era of printed tables, not a result. At n = 30 the t multiplier is 4.35% larger than z, which is small and is not nothing, and a computer does not care which it looks up.

Where σ genuinely is known: a manufacturing process with years of calibration data, a measuring instrument with a documented precision, a simulation where you set it. Those are real, and they are rare.

Where the z-test is exactly right

Proportions. And the reason is structural rather than a matter of sample size.

For a binary outcome the variance is not a free parameter. It is p(1−p) — completely determined by the proportion itself. There is no separate spread to estimate.

So when the null hypothesis says “the rate is 8%”, it has also said the variance is 0.08 × 0.92. Nothing is estimated, the criticism of the mean case evaporates, and the standard normal is the right reference distribution.

Note the detail this implies, which the tool follows: the one-proportion test uses p₀ in the standard error, not p̂. Under the null, p₀ is the true value, so it is the better estimate of the variance. Using the observed rate instead is a common and avoidable error.

The two-proportion test does the same thing differently. Under its null the two rates are equal, so the best estimate of that shared rate is the pooled one, (x₁+x₂)/(n₁+n₂), and that is what goes in the standard error.

This is why A/B tests are conventionally z-tests. The outcome is binary, so the variance follows from the rate, and it is the correct test rather than a large-sample approximation to something better.

The pooled standard error, and why it is not what you would guess

A subtlety worth pausing on, because it looks like an inconsistency until you see the logic.

For the two-proportion test the standard error uses the pooled rate. For a confidence interval on the difference between two proportions, it does not — it uses each group’s own rate.

That is not an inconsistency, it follows from what each is doing. A test computes everything assuming the null is true, and the null says the two rates are the same. Under that assumption, pooling gives the best estimate of the single shared rate, and using it makes the test more powerful.

An interval assumes nothing. It has to describe the difference whatever it is, so each group supplies its own variance.

The practical consequence: a two-proportion test and a confidence interval for the same difference can disagree marginally at the boundary — the interval can just include zero while the test just rejects, or the reverse. It is a known and accepted feature of using different standard errors for different jobs, and it only shows up in borderline cases.

When the normal approximation stops working

The z-test for proportions is an approximation: it treats a discrete binomial count as though it were continuous and normal. That works well in the middle and badly at the edges.

The standard condition is at least 10 successes and 10 failures in each group. Some texts say 5, some say 15; the important thing is that both counts matter, not just the sample size. A group of 10,000 with only 3 successes fails the condition badly despite being large.

The tool checks the smallest of the four cells and warns when it falls short, because that is the binding constraint and it is easy to overlook when the totals look comfortable.

When the condition fails, the alternative is Fisher’s exact test, which computes the probability directly from the hypergeometric distribution and makes no normal approximation at all. It is exact at any cell count and is the right choice for small or lopsided tables.

A continuity correction — shrinking the difference by 1/(2n) before dividing — is sometimes applied to improve the approximation. It makes the test more conservative and is somewhat out of fashion, on the grounds that if the counts are small enough for it to matter you should use an exact test instead.

Cohen's h, and why percentage points mislead

A difference of 2 percentage points means very different things depending on where it sits, and this is the standard trap in reading A/B test results.

Going from 1% to 3% is a tripling. Going from 50% to 52% is a 4% relative change. Both are “2 points”, and they are not remotely the same result.

Cohen’s h handles this by working on the arcsine-transformed proportions: h = 2·arcsin√p₁ − 2·arcsin√p₂. The transformation stretches the ends of the scale, so a shift near 0 or 1 counts for more than the same shift in the middle — which matches how the difference actually behaves.

The conventional labels mirror Cohen’s d: 0.2 small, 0.5 medium, 0.8 large. As with d, they are a fallback for when you have no domain yardstick, and a domain yardstick is always better. In a conversion funnel, whether a 0.3-point lift pays for the engineering is a business question, not a statistical one.

Report all three: the two raw rates, the difference in points, and the relative change. Each hides something the others show.

Using this for an A/B test

The two-proportion z-test is the standard analysis for a conversion experiment, and the arithmetic is the easy part.

Fix the sample size in advance. Checking the p-value as results arrive and stopping when it dips below 0.05 inflates the false positive rate substantially — with enough peeking you will cross the line eventually even when the variants are identical. If you need to monitor continuously, that requires a sequential design built for it, not repeated use of this test.

One metric, decided beforehand. Testing signup rate, then click rate, then revenue per visitor, and reporting whichever moved is running three tests and reporting one.

Check the randomisation actually worked. Wildly unequal group sizes usually indicate a bug in assignment or logging rather than chance, and a broken assignment invalidates the test however clean the arithmetic looks.

Watch the novelty effect. A change often produces a short-lived response simply because it is different. A test that runs a few days can measure the novelty rather than the change, which is why full business cycles are the convention.

And the point that outlives the statistics: significance is not impact. A result significant at p = 0.001 with a 0.05-point lift is real and probably not worth shipping. Size the test around the smallest lift that would change your decision, and read the interval rather than the p-value.

The z-test and chi-square give the same answer

A two-proportion z-test and a chi-square test of independence on the same 2×2 table are the same test. Not similar — the same.

The chi-square statistic is exactly z², and the p-values match to the last digit, because chi-square with 1 degree of freedom is the distribution of a squared standard normal. This site’s distribution suite asserts that identity directly.

So the choice between them is presentational, with one real asymmetry: the z-test can be one-tailed and chi-square cannot. Squaring destroys the sign, so a chi-square test is inherently two-sided. If the direction matters and was fixed in advance, use z.

Chi-square earns its place when the table is larger than 2×2 — three or more groups, or a categorical outcome with several levels — where there is no single difference to take a z of.

One practical consequence: if a colleague’s chi-square and your z-test disagree on the same 2×2 data, one of you has made an arithmetic error or applied a continuity correction the other did not. They cannot legitimately differ.

Sources and methodology

References for the approximation conditions and the pooled standard error.

• NIST/SEMATECH e-Handbook — comparing proportions, including the sample-size conditions
• Agresti & Coull on approximate versus exact inference for binomial proportions — where the normal approximation holds and where it does not
• R documentation for prop.test() — the reference behaviour for the pooled standard error and the continuity correction
• t-test Calculator — the test you probably want if you are comparing means
• p-value Calculator — what the p from this test can and cannot support
• How we calculate — methodology

Method. The one-proportion test uses p₀ in the standard error rather than p̂, because the null fixes the variance; the two-proportion test uses the pooled rate, for the same reason applied to a shared null. Tail probabilities come from a survival function rather than 1 − CDF, so a large z does not underflow to p = 0 — the normal upper tail here is still accurate at z = 30, where the subtraction form returns a hard zero. The mean mode computes the t-test on the same inputs and prints both, because the difference between them is the entire content of the assumption being made. The suite asserts the pooled standard error against a hand computation, that a two-tailed p is exactly twice the one-tailed p, and — in the distribution suite — that chi-square with 1 degree of freedom is the square of a standard normal, which is why this test and a 2×2 chi-square cannot legitimately disagree. That engine is verified on every change against 155 assertions. The count and the per-case breakdown are published on the formula verification page.

Related calculators

Where this goes next:

More in Math, or browse all calculators.