Fisher’s Exact Test Calculator

Three two-sided p-values, and no ordering between them

The one-sided p-value is unambiguous: sum the probability of every table at least as far in one direction. The two-sided version requires deciding what “as extreme in either direction” means, and the hypergeometric is not symmetric, so the decision matters.

Minimum likelihood sums the probability of every table no more probable than the one observed. It is R’s fisher.test default and what most textbooks describe.

Doubling takes the smaller one-sided tail and multiplies by two. It is what a hand calculation does, and what several software defaults use. It assumes a symmetry the distribution does not have, so it can exceed 1 before capping.

Blaker’s method takes the observed tail and adds the largest opposite tail that does not exceed it. It is the only one of the three that yields nested confidence intervals, which is a real advantage when you want the interval and the test to agree.

On [[5,1],[2,7]] they give 0.0406, 0.0699 and 0.0406. Two significant, one not. The table is not contrived; it is an ordinary small comparison, and the disagreement is the kind that decides what a paper concludes.

And there is no total order between them. Across 5,000 random tables only one inequality held every time: Blaker is never larger than doubling, which is Blaker’s own theorem. Blaker exceeds minimum likelihood on about 0.6% of tables, and doubling falls below minimum likelihood on about 1.5% — so neither pairing has a reliable direction. You cannot know which is the conservative choice without computing all three.

The test is conservative, and by exactly how much

“Fisher’s exact test is conservative” is repeated everywhere and almost never quantified. It can be, precisely, because the space of possible outcomes is finite.

With the margins fixed there are only a handful of tables. At 10/10/10 margins there are eleven, and they carry only six distinct p-values. The test cannot produce a p-value it does not have.

So the actual size is a finite sum, not a simulation. Add the probability of every table that would be rejected at a nominal 5%, and at 10/10/10 margins the answer is 2.3014%. Fewer than half the false positives the level advertises.

The reason is a gap in the available p-values. At those margins they run 0.000011, 0.001093, 0.023014, then jump straight to 0.178895. There is nothing between 2.3% and 17.9% to spend the rest of α on.

It gets worse on smaller tables. At 8/8/8 margins the actual size is 1.0101% — one fifth of nominal. And it does not vanish with sample size the way an approximation error would: at 50/50/50 it is still only 2.7312%.

Conservative means less powerful, not safer. A test spending 2.3% of a 5% budget is leaving real effects undetected to buy a guarantee it was never asked for. That is the trade conditioning on the margins makes, and it is why Barnard’s unconditional test — which does not condition, and is more powerful — exists as an alternative for designs where the margins were not actually fixed in advance.

The tea tasting, and why the design had four of each

The test has an origin story that is also its clearest worked example, and the design detail in it is the part usually left out.

Muriel Bristol claimed she could tell whether the milk went in before the tea or after. Fisher designed the experiment: eight cups, four each way, and she was told in advance that it was four and four.

She identified all eight correctly. The probability of that by chance is 1 in C(8,4) = 70, or 0.014286 — and the tool returns exactly 1/70, because the point probability of that table is the whole one-sided tail. Nothing is more extreme than being completely right.

Telling her the split is what makes the margins fixed. Both row totals and both column totals are known before a single cup is tasted, which is the condition the hypergeometric derivation needs. Without it she could have said “milk first” to all eight, and the analysis would be a different one.

That is the rare case where the conditioning is genuinely justified. In most 2×2 tables the margins are not fixed by design — you did not decide in advance how many patients would recover. Fisher’s test conditions on them anyway, which is the assumption the conservativeness comes from, and the main reason the unconditional alternatives are argued for.

Seven of eight would not have been enough. Getting three of four right on one side gives a one-sided p of 17/70 = 0.243. With four cups each way, only a perfect score clears 5% — which means Fisher designed an experiment with exactly one passing outcome, and said so.

Against chi-square, and against Yates

The standard advice is to use chi-square unless an expected count falls below 5, and then switch. The threshold is a rule of thumb, and both halves of the advice are shakier than they sound.

Chi-square approximates a discrete count by a continuous distribution. On small tables that approximation runs anti-conservative — it returns a p-value smaller than the truth, so it finds significance the data does not support.

Yates’s continuity correction pushes back, and usually too hard. It subtracts 0.5 from each absolute deviation before squaring, which always lowers the statistic. The result is a test that is typically more conservative than Fisher’s, and it has been criticised for decades as over-correcting. The tool prints both so the size of each adjustment is visible.

The 5 threshold has no theory behind it either. It is Cochran’s rule of thumb from 1954, and modern work suggests the approximation is often acceptable with smaller expected counts and sometimes poor with larger ones. Where an exact test is available there is no reason to be near the boundary at all.

On any modern machine, exact is free. The argument for chi-square on a 2×2 was computational, and the computation is a handful of binomial coefficients. The remaining argument is that chi-square does not condition on the margins — which is a real argument, and points at Barnard’s test rather than back at chi-square.

For larger tables the exact test becomes expensive fast. Beyond 2×2 the enumeration grows sharply, and the chi-square test of independence is the practical tool. This page is about the case where the exact answer is cheap and the approximation is worst.

The odds ratio, and the one Fisher’s test is actually about

The tool prints ad/bc, which is the sample odds ratio. It is not quite the quantity the test is built around, and the gap shows up on exactly the tables where you most want a number.

Fisher’s test is a test of the CONDITIONAL odds ratio, estimated by conditional maximum likelihood rather than by ad/bc. The two are close on large tables and diverge on small ones, so R’s fisher.test reports a different number from the one you get by hand.

A zero cell sends ad/bc to zero or infinity. The p-value stays perfectly finite — the tea-tasting table has two zero cells and a p of 0.0286 — but the odds ratio does not, and the usual patch of adding 0.5 to every cell changes the estimate by an amount that depends on the patch.

The odds ratio is not the relative risk, and on common outcomes the two diverge sharply. An odds ratio of 6 can correspond to a relative risk of 1.5. The odds ratio calculator computes both alongside the risk difference and the number needed to treat.

A p-value and an effect size answer different questions. On a small table the p-value can be large while the odds ratio is enormous — because three observations cannot rule out chance, not because the effect is small. Report both, and the interval.

And the interval should match the test. If you quote Blaker’s p-value, quote Blaker’s interval: it is the only one of the three whose intervals nest, so a 99% interval actually contains the 95% one. Mixing a minimum-likelihood p with a different interval is how a paper ends up with a significant test and an interval containing 1.

When conditioning on the margins is the wrong model

The exactness comes from treating both margins as fixed. That is a real assumption about how the data arose, and in most studies it is false.

Three sampling designs produce a 2×2 table. Both margins fixed, as in the tea tasting. One margin fixed, as in a trial with 50 per arm. Neither fixed, as in observing whoever walks through the door for a month.

Fisher’s test conditions on both regardless. Where only one margin was fixed by design, conditioning on the other discards information — and that lost information is exactly the conservativeness the previous section measured.

Barnard’s test does not condition. It maximises over the nuisance parameter instead, is uniformly more powerful for the one-margin-fixed design, and is computationally heavier. For a small trial it is the better-matched analysis, and it remains far less used.

The argument has been running since the 1940s. Fisher’s position was that conditioning removes a nuisance parameter and is therefore the right inference; his critics held that it throws away real information about a design that did not have fixed margins. It is not settled, and a tool that presents Fisher’s test as simply “the exact test” is taking a side silently.

In practice the difference matters most where the data is smallest — which is the only place anyone reaches for an exact test. With large counts every method agrees, and chi-square agrees with them too.

Sources and methodology

References for the test, its conventions and the argument about conditioning.

• Fisher, “The Logic of Inductive Inference”, JRSS 98(1), 1935 — where the tea-tasting design appears
• Blaker, “Confidence Curves and Improved Exact Confidence Intervals for Discrete Distributions”, Canadian J. Statistics 28(4), 2000
• Agresti, “A Survey of Exact Inference for Contingency Tables”, Statistical Science 7(1), 1992 — the conditioning argument, surveyed
• Chi-Square Calculator — the approximation this replaces, and what it costs on small tables
• Odds Ratio Calculator — the effect size to report beside the p-value
• How we calculate — methodology

Method. Point probabilities come from the hypergeometric distribution computed in log space, so a table with large margins does not overflow. All three two-sided conventions are computed on the same enumeration, with a 1e-7 relative slack on the comparisons because exact probability ties are common on symmetric margins and float equality would drop half of them. The exact size of the test is obtained by enumerating every table the margins permit rather than by simulation. The suite pins the tea-tasting probability at exactly 1/70, confirms the point probabilities against a direct factorial computation, establishes that only Blaker ≤ doubling holds universally across 5,000 tables while the other two orderings each fail, and checks that the actual size never exceeds the nominal level across 48 margin sets while falling short of it by up to 5 percentage points. That engine is verified on every change against 51 assertions. The count and the per-case breakdown are published on the formula verification page.

Related calculators

Where this goes next:

More in Math, or browse all calculators.