Sign Test Calculator

What discarding the magnitudes costs, in numbers

Every paired test faces the same data and uses a different amount of it. The sign test uses the least.

Twelve patients, every one improved: p = 0.000488. That is 2 × 0.5¹², the smallest p-value twelve pairs can produce, and it is reached whether the improvements were 1 unit or 100.

Change one patient to a 1-unit worsening and p becomes 0.006348. Thirteen times larger, from a change that moved the mean difference by less than a point. The other eleven improvements were large; the test never saw their size, so it could not weigh them against the reversal.

Against a paired t-test on normal differences, the gap is measurable. Across 1,500 simulated studies of 20 pairs with a real shift, the t-test detected it 39.5% of the time and the sign test 25.9% — a third of the study’s power, given away.

That is the correct price for the assumption it drops. The t-test needs the differences to be roughly normal; Wilcoxon needs them symmetric about their median. The sign test needs neither, and on data where those assumptions fail it is the one still answering the question asked.

And the invariance is total. Replace every difference with any other number of the same sign and the p-value does not move. Apply a log, a square root, a rank transform — anything monotone, to both columns — and the answer is identical. Verified across 500 random studies.

Ties shrink your study, quietly

A pair that did not change gives no evidence about direction, so the classical procedure drops it. The consequence is larger than it sounds.

Ten pairs with five ties return the p-value of a five-pair study. Not an adjusted ten-pair study — a five-pair one, exactly. The tool prints 0.0625 for both, because they are the same binomial calculation.

And 0.0625 is the smallest a five-pair study can ever produce. Five pairs all moving the same way still cannot reach 0.05. Half the sample vanished and took the possibility of a significant result with it.

The tool reports the floor for your surviving pair count and warns when it sits above 0.05. That is a design fact, available before any data is collected, and it is the single most useful thing to know about a small paired study.

Ties are common in exactly the data the sign test suits. Ordinal scales, rounded measurements, and preference tasks with a “no difference” option all generate them. If a third of your pairs tie, your effective n is two thirds of what you planned.

Alternatives exist and none is standard. Splitting ties evenly between the two directions, or counting them against the alternative, both change the answer and neither is the classical test. This tool discards, reports the count, and leaves the decision visible rather than buried in a default.

Exact, not approximate — and why that matters here

Many implementations switch to a normal approximation above some sample size. This one never does, and the reason is that the sign test is used precisely where approximations are worst.

The p-value is summed from the binomial directly. 2 × Σ C(m, i) × 0.5^mfor i from 0 to the smaller count. The suite checks this against an independently summed binomial across all 350 (m, k) combinations up to m = 25 — every one an exact match.

A normal approximation is poor at the sample sizes that matter. At m = 8 with 7 increases, the exact p is 0.0703; the normal approximation without continuity correction gives 0.0339, which crosses the conventional threshold in the wrong direction.

Exactness also means the test is conservative, and measurably so. Because binomial p-values are discrete, the achievable values jump: at m = 15 no outcome gives a p between 0.0352 and 0.1185 — the whole neighbourhood of 0.05 is unreachable. Under a true null the realised rejection rate at α = 0.05 came out at 3.33% rather than 5% across 4,000 simulated null studies.

That conservatism is a guarantee, not a flaw. An exact test promises never to exceed its stated error rate, and the price is sometimes falling short of it. The suite verifies the promise on 4,000 simulated null studies.

It tests the median, not the mean

The null hypothesis is that the median of the differences is zero — that a randomly chosen pair is as likely to go up as down. That is not the same claim as the t-test’s.

The t-test’s null is that the mean difference is zero. On symmetric differences the two coincide. On skewed differences they do not, and the tests can disagree without either being wrong.

Try the skewed preset. Thirteen pairs improve modestly and one improves enormously. The mean difference is dominated by the single large change; the median is not. Both answers are correct answers to different questions.

Which question you want is a substantive decision. “Does the typical person improve” is a median question. “Does total output rise” is a mean question, because totals are means multiplied by n. The statistics cannot choose for you.

The tool prints both the median and the mean difference beside their respective p-values, so the disagreement shows up as two numbers rather than as a puzzle.

Sign test, Wilcoxon or paired t-test

Three tests on the same data, in increasing order of how much they assume and how much power they extract.

The paired t-test uses the actual magnitudes. Most powerful when the differences are roughly normal, and it is fairly robust to mild departures because the central limit theorem applies to the mean difference. It is vulnerable to outliers, which move both the mean and the standard error.

Wilcoxon signed-rank uses the ranks of the magnitudes. It keeps the information that one change was larger than another while discarding how much larger, which buys most of the t-test’s power with far less sensitivity to extremes. It assumes the differences are symmetric about their median — an assumption that is easy to forget and fails on skewed data.

The sign test uses only the direction. It assumes nothing beyond independent pairs, and it is the only one of the three still valid when the differences are badly skewed or when the outcome is ordinal and the differences have no meaningful size at all.

Choose before looking at the data. Running all three and reporting the smallest is an uncorrected multiple comparison. The tool shows two of them together for understanding, not for selection.

The honest default for paired ordinal data is the sign test. For continuous data with no visible outliers or skew, it is the t-test. Wilcoxon sits between them and is the right answer more often than either — provided its symmetry assumption actually holds.

Reporting a sign test

Four things belong in the sentence, and two of them are usually missing.

Report the split, not just the p-value. “11 of 12 pairs decreased” is the finding; the p-value is a summary of it. A reader can check the arithmetic from the split and cannot from the p-value alone.

Report the ties separately. “n = 20” when eight pairs tied is misleading, because the test ran on twelve. State both numbers.

Say the test was exact. Exact and approximate sign tests give different p-values at the sample sizes where sign tests are used, and readers cannot tell which they are looking at.

Give an effect size or a confidence interval. A p-value from a sign test carries no information about how large the change was — by construction. The median difference, and ideally a confidence interval around it, is what tells the reader whether the change matters.

And if the result is not significant, check the floor. With few usable pairs the test may have had no chance of significance at any effect size, which is a different finding from “no effect detected” and should be reported as one.

Sources and methodology

References for the sign test and its place among paired procedures.

• Dixon & Mood, “The Statistical Sign Test”, JASA 41(236), 1946
• Wilcoxon, “Individual Comparisons by Ranking Methods”, Biometrics Bulletin 1(6), 1945 — the test that uses the ranks instead
• Putter, “The Treatment of Ties in Some Nonparametric Tests”, Annals of Mathematical Statistics 26(3), 1955
• Wilcoxon Signed-Rank Calculator — the middle option, using ranks of the magnitudes
• T-Test Calculator — the paired test that uses the magnitudes in full
• How we calculate — methodology

Method. The p-value is summed exactly from the binomial rather than approximated, at every sample size, and the suite verifies it against an independently computed binomial across all 350 combinations of m up to 25 and every possible count within them. Ties are discarded in the classical manner and the suite checks that a ten-pair study with five ties returns exactly the p-value of the corresponding five-pair study. Calibration is checked on 4,000 simulated null studies, where the realised rejection rate came out at 3.33% — at or below the nominal 5%, as an exact test must be; the power cost against a paired t-test was measured on 1,500 studies (39.5% against 25.9%) rather than quoted; and monotone invariance is verified across 500 more. That engine is verified on every change against 86 assertions. The count and the per-case breakdown are published on the formula verification page.

Related calculators

Where this goes next:

More in Math, or browse all calculators.