A p-value built from your own numbers, not from a distribution.
Every split, counted
Every one of the 184,756 ways to split these twenty values into two groups of ten is enumerated. Exactly 10 of them give a difference as extreme as the observed −6.2000, so p = 10/184,756 = 0.00005413 — a ratio of two integers.
all 184,756 splits enumerated
p = 0.00005413
10 of the 184,756 possible splits give a difference at least as extreme as the observed -6.2000. That fraction IS the p-value — a ratio of two integers, with no distribution assumed anywhere. A Welch t-test on the same data gives 1.0242e-5.
Permutation p
0.00005413
exact — every split counted
Welch t-test p
1.0242e-5
for contrast
Observed difference
-6.2000
14.300 − 20.500
Splits
184,756
C(20, 10)
The null distribution, built from your own data
Nothing here was assumed. Every point in that curve is an actual rearrangement of your twenty-odd numbers, and the p-value is the share of it beyond the mark.
One-sided and two-sided permutation p-values with their counts
Test
p
Splits counted
Two-sided
0.00005413
10 of 184,756
Group 1 greater
0.99999459
184,755 of 184,756
Group 1 less
0.00002706
5 of 184,756
Smallest p this design can produce
0.00001083
2 of 184,756
No normality assumption anywhere Exact below 400,000 splits, sampled above Assumes exchangeability, which is not nothing
What this tool shows
Two groups of ten can be split 184,756 ways, and this tool counts every one of them in about 80 milliseconds. On the shipped example exactly 10 of those splits produce a difference at least as extreme as the observed one, so p = 10 ÷ 184,756 = 0.00005413 — a ratio of two integers, with no normal distribution, no t distribution and no approximation anywhere in it.
Exhaustive enumeration of every split below 400,000, and sampling above it
The null distribution plotted from your own data, with the observed value marked
Two-sided and both one-sided p-values, with the split counts behind them
The smallest p-value the design can possibly produce — the granularity floor
A Welch t-test on the same data, for contrast
Monte Carlo error reported when the test samples rather than enumerates
Exact enumeration No distribution assumed Null distribution shown Granularity floor
Exchangeability is the assumption, and it is not nothing.
Updated 12 September 2026 · Works in any browser, no installation
A permutation test asks how often a random rearrangement of your data would produce a difference as large as the one you observed. If the two groups are interchangeable under the null, every split of the pooled values is equally likely — so counting the extreme ones gives a p-value directly. No distribution is assumed, which is what makes it exact rather than approximate, and it is the logic behind Fisher’s exact test applied to continuous data.
At a glance
Formula shown
Pool the n₁ + n₂ observations. For every way of choosing n₁ of them — C(n₁+n₂, n₁) ways — compute the difference in group means. The two-sided p is the proportion of those differences whose absolute value is at least the observed one. With n₁ = n₂ = 10 that is C(20,10) = 184,756 splits; with 15 and 15 it is 155,117,520, which is why the tool switches to sampling above 400,000.
Scenario support
Small samples where normality is doubtful, heavily skewed outcomes, data with one or two extreme values, comparing medians or trimmed means where no standard test exists, A/B tests with few conversions, and any two-group comparison where a reviewer will ask what the t-test assumed.
Educational estimate
Planning support from the values you enter — not professional advice.
Exact means every split, not a good approximation
The word gets used loosely. Here it is literal, and the tool reports which of the two regimes it is in.
Below 400,000 splits the test enumerates all of them. The p-value is then a count divided by a count, and running the tool twice on the same data gives the identical answer to every digit.
On the shipped example that is 10 out of 184,756. Not “approximately 0.00005” — exactly 10/184,756 = 0.0000541254.
Above the limit it samples, and says so, reporting the Monte Carlo error alongside the estimate. Fifteen against fifteen is already 155 million splits, so sampling is not an edge case.
A sampled permutation p-value is still valid, just noisy: 20,000 samples give an interval of roughly ±0.003 around a p near 0.05, which matters when a decision sits on the boundary.
A small design has a smallest possible p-value
This is the property that surprises people, and it is visible in the table rather than hidden.
With five against five there are only 252 splits. The most extreme arrangement is one of them, and its mirror is another, so the smallest achievable two-sided p is 2/252 = 0.007937.
No arrangement of five against five can ever produce p < 0.007937. Not because the effect is small, but because the design cannot resolve any finer.
Four against four is 70 splits, so the floor is 0.028571 — which means a four-against-four comparison can only just clear 0.05, however perfectly separated the groups are.
Three against three cannot reach 0.05 at all. Twenty splits, a floor of 0.1. A perfectly separated result is not significant, and no amount of effect size changes that.
The tool prints that floor so a non-significant result from a tiny design can be read as “this design could not have found it” rather than “there is nothing there”.
Against the t-test, and where each one wins
The two agree closely on well-behaved data and part company exactly where the t-test’s assumptions are strained.
On the shipped ten-against-ten example the permutation p is 0.00005413 and the Welch t gives 1.0242×10⁻⁵. Close enough that either supports the same conclusion, which is what you expect from roughly normal data.
The t-test is more powerful when its assumptions hold, and not by much. Giving up normality costs surprisingly little power on samples this size.
The permutation test wins when one value dominates the standard error. The tool’s second preset has a single value of 60 among numbers between 4 and 9; the t statistic is crushed by the inflated variance while the permutation test simply counts.
And it applies to statistics no t-test covers. A difference in medians, in trimmed means, in interquartile ranges — any statistic at all can be permuted, because the procedure never needs to know its sampling distribution.
Exchangeability is the assumption, and it is a real one
“Assumption-free” is the usual selling point and it is not quite right. The distributional assumption is gone; one structural assumption remains.
Under the null, every observation must be equally likely to have come from either group. That is exchangeability, and it is what licenses treating all 184,756 splits as equally probable.
It fails when the groups differ in something other than the effect. If group 2 was measured with a different instrument, or later in time, or by a different observer, the values are not interchangeable even under a null hypothesis about means.
The sharp null is what is actually being tested: that the treatment had NO effect on any unit, not merely that the mean effect is zero. A treatment that raises some values and lowers others by the same amount violates the sharp null while leaving the means alone.
Unequal variances are a live issue too. When the groups differ in spread but not in location, exchangeability fails and the test can reject — correctly, since the distributions genuinely differ, but not for the reason a reader will assume.
The paired version permutes signs, not labels
A design with matched pairs has a different exchangeability structure, and permuting group labels would be wrong.
For paired data the unit is the difference, and the null says its sign is arbitrary — so the permutations are the 2ⁿ assignments of plus and minus.
Ten pairs is 1,024 sign patterns, so the floor on a two-sided p is 2/1024 = 0.001953, and twenty pairs is 1,048,576.
That is the exact test behind the Wilcoxon signed-rank test, applied to the raw differences instead of their ranks.
This tool runs the two-sample version. Feeding it paired data as two columns discards the pairing, which throws away exactly the information the design was built to capture.
Reporting a permutation test
Four things, and the first is what separates an exact result from an estimate.
Say whether the test enumerated or sampled. An exact p-value and a 20,000-sample estimate of the same quantity are different claims, and only one of them is reproducible to the last digit.
Give the statistic you permuted. A difference in means, in medians and in trimmed means give different p-values on the same data, and the procedure does not care which you chose.
Report the number of splits. It sets the granularity floor, and a reader needs it to know whether a non-significant result was even reachable.
And state what exchangeability means for your design. It is the one assumption left standing, and it is a claim about how the data was collected rather than about its distribution.
Sources and methodology
References for permutation tests and their exactness.
Method. Splits are enumerated by a recursive walk over index combinations that carries a running sum, so no permutation array is ever materialised and 184,756 splits complete in about 80 milliseconds. The exactness claim is structural rather than statistical: the p-value is a count divided by C(n₁+n₂, n₁), and the suite asserts that the enumeration produces exactly that many splits at a range of group sizes. Above 400,000 splits the test samples with a Fisher-Yates shuffle on a mulberry32 stream and reports the Monte Carlo error rather than presenting the estimate as exact. Comparisons use a tolerance of 1e-12 so that a split identical to the observed one is counted as at least as extreme rather than lost to floating point. A Welch t-test is computed on the same data for contrast. Fewer than two values in either group, and any non-finite value, return no result. That engine is verified on every change against 95 assertions. The count and the per-case breakdown are published on the formula verification page.
Related calculators
Where this goes next:
t-testOne-sample, two-sample and paired t-tests defaulting to Welch, with Student's pooled version printed beside it — and a warning when the two disagree on the verdict.
Fisher's Exact TestThe exact p for a 2×2 table under all three two-sided conventions, because they disagree — 0.0406 against 0.0699 on the built-in table, across the 5% line — plus the test's actual size by enumeration, which is 2.30% at a nominal 5%.
Bootstrap Confidence IntervalNormal, percentile, basic and BCa intervals from one seeded resample set, with measured coverage: on lognormal data at n = 20 a nominal 95% achieves 82.60% to 87.20% depending on the method.
Mann-Whitney UMann-Whitney U with the tie correction and both medians shown, because it is not a test of medians — two samples with identical medians of 4 are separated at p = 0.01.
Wilcoxon Signed-RankReports how many zero differences it dropped and gives the Hodges-Lehmann shift, because the classical and Pratt variants disagree on the same data and the median of the differences is not what this test estimates.
p-valueA p-value from a t or z statistic, one- or two-tailed — with a panel that holds an effect fixed and grows the sample, so you can watch significance appear from nothing but n.
An educational tool. A permutation test drops the normality assumption but not exchangeability — under the null every observation must be equally likely to have come from either group, which fails if the groups differ in measurement, timing or spread as well as in the effect being tested.
Published a two-sample permutation test that ENUMERATES every split when there are fewer than 400,000 of them — two groups of ten is C(20,10) = 184,756, completed in about 80 milliseconds by a recursive walk that never materialises a permutation array.
Made 'exact' literal: on the shipped example exactly 10 of those 184,756 splits give a difference at least as extreme as the observed −6.2, so p = 10/184,756 = 0.0000541254 — a ratio of two integers, with no normal or t distribution anywhere in it.
Printed the granularity floor, which is the property that surprises people. Three against three has 20 splits so its smallest achievable two-sided p is 0.1: a perfectly separated result cannot be significant, however large the effect. Four against four floors at 0.028571 and five against five at 0.007937.
Plotted the null distribution built from the user's own numbers, and reported Monte Carlo error explicitly when the design is too large to enumerate rather than presenting a sampled estimate as exact.
Stated the one assumption that survives — exchangeability — as a claim about how the data was collected rather than about how it is distributed.
Add this calculator to your site
Responsive embed — and private: nothing your visitors type leaves their browser.