p-value Calculator

What the number literally is

Read the definition slowly, because every clause is doing work.

The probability of observing data at least as extreme as yours, assuming the null hypothesis is true.

“Assuming the null is true” makes it a conditional probability, and the condition is the hypothesis you were hoping to reject. You have computed P(data | H₀). That is not P(H₀ | data), and the two can differ by orders of magnitude — confusing them is a formal fallacy with a name, the transposed conditional.

“At least as extreme” means the p-value includes outcomes you did not observe. It is a tail area, not the probability of your particular result, which for continuous data is zero.

So a p of 0.03 says: if there were genuinely no effect, data this striking would turn up about three times in a hundred. That is all it says. Whether there is an effect depends on that, on how plausible an effect was beforehand, on how the study was run, and on how many other things were tested.

The four things a p-value is not

These are misread so consistently that the American Statistical Association published a formal statement about it in 2016 — the first time in its history it had issued guidance on a specific statistical practice.

Not the probability the null hypothesis is true. p = 0.03 does not mean a 3% chance there is no effect. That number depends on how likely an effect was before you collected anything, which a p-value never sees.

Not the probability your result is a fluke. Same error, differently worded. A p-value assumes the null and asks about the data; it cannot turn round and give you a probability about the hypothesis.

Not a measure of effect size. This is the one the tool above demonstrates. A trivial effect measured on a large enough sample gives a tiny p. A large, important effect on a small sample gives a large one. p answers “could this be noise?” and never “is this big?”.

Not evidence of no effect when it is large. p = 0.4 means you did not find enough evidence, which is compatible with there being no effect and equally compatible with there being a real one you were underpowered to detect. Absence of evidence is not evidence of absence.

Why enough data makes anything significant

The table in the tool holds an effect at exactly a fifth of a standard deviation — what Cohen called small — and changes nothing but the sample size.

At n = 10 the p-value is 0.54: nothing to report. At n = 100 it is 0.048, and would be written up as significant. At n = 2500 it is about 4 × 10⁻²³ — a hundred million times past the five-sigma threshold particle physics uses to announce a discovery.

The effect was identical in all three. What changed was precision. The test statistic grows with the square root of n, so any effect that is not exactly zero will eventually cross any threshold you pick, given enough data.

Two consequences worth holding onto. In a very large study — a platform A/B test on millions of users, say — statistical significance is nearly free and tells you almost nothing; the question worth asking is whether the effect is large enough to act on. And in a small study, failing to reach significance is unsurprising and is not evidence that the effect is absent.

This is why effect sizes and confidence intervals are increasingly required alongside p-values, and why this page shows Cohen’s d beside p rather than in a footnote.

The tail choice, and the free significance it can buy

A two-tailed p is exactly twice the one-tailed p on the same side. The tool prints both so the factor of two is visible.

Two-tailed asks whether there is a difference in either direction. It is the default, and it is correct whenever a change in either direction would be interesting — which is most of the time.

One-tailed asks only about one direction, and puts the whole rejection region on that side. It is legitimate when a change the other way would be meaningless or impossible to act on, and when the direction was committed to before the data existed.

That last condition is the whole difficulty. Looking at the data, seeing which way it went, and then choosing the matching one-tailed test halves your p-value without collecting a single extra observation. A result at p = 0.08 two-tailed becomes p = 0.04 one-tailed, and crosses the line. Nothing about the evidence changed.

The practical rule: use two-tailed unless you can point to something written down beforehand that fixed the direction. A one-tailed test chosen after the fact is not a stricter analysis, it is a weaker one wearing a stricter label.

Where 0.05 came from, and what it costs

0.05 is a convention with an author and no derivation. Ronald Fisher suggested it in 1925 as a convenient round figure, explicitly as a rough guide rather than a standard, and it hardened into one.

What it means operationally: if the null is true, you will reject it 5% of the time anyway. That is the false positive rate you have agreed to accept, and it is a choice about the cost of being wrong in each direction rather than a fact about evidence.

The costs are visible in fields that need more certainty. Particle physics uses five sigma, about 0.0000003, because a false discovery is enormously expensive. Genome-wide association studies test hundreds of thousands of positions and correct accordingly, because at 0.05 they would report thousands of false hits by construction.

The multiple comparisons problem follows directly. Run twenty independent tests at 0.05 and there is a 64% chance at least one comes back significant with nothing there at all. Run a hundred and it is over 99%. This is why the number of tests performed — including the ones not reported — changes what a single p-value means, and why corrections such as Bonferroni exist.

It is also why a p just under the line deserves less confidence than it usually gets. p = 0.049 and p = 0.051 are the same evidence. Only one of them gets published.

How p-values get manufactured

Not by fraud, mostly. By ordinary, defensible-looking decisions taken after the data arrives.

Optional stopping. Checking the p-value as data comes in and stopping when it drops below 0.05. This inflates the false positive rate dramatically — with enough peeking you will eventually cross the line even when nothing is there, because the p-value wanders.

Choosing the analysis afterwards. Trying several outcome measures, or several ways of excluding outliers, and reporting the version that worked. Each attempt is a test; only one is reported.

Subgroup hunting. No overall effect, but an effect among women over 50 in the second cohort. With enough subgroups something always appears.

HARKing — hypothesising after the results are known — then presenting the finding as though it had been predicted. This converts an exploratory result into a confirmatory-looking one, and the p-value attached to it no longer means what it claims.

The defences are procedural rather than statistical: decide the analysis before seeing the data, pre-register it where you can, report every test you ran, and treat anything found by exploration as a hypothesis to test on new data rather than a result.

What to report alongside it

Nobody serious argues for abolishing p-values. The argument is that a p-value alone is not a result.

A confidence interval. It contains the p-value’s information — if the null value is outside the 95% interval, p is below 0.05 — and adds what p omits: the size of the effect and how precisely it was measured. “The difference was 3.2 points, 95% CI 0.4 to 6.0” says everything “p = 0.03” says and more.

An effect size. Cohen’s d, a risk ratio, a percentage change — something in units a reader can judge. This is the quantity that does not move with sample size.

The sample size and the power. A non-significant result from a study with 20% power is almost uninformative, and the reader cannot know that without being told.

How many tests were run. One p of 0.03 out of one test is not the same evidence as one p of 0.03 out of forty.

The honest short form: report the estimate, its interval, and the p-value — in that order of prominence.

Sources and methodology

The formal statements behind the cautions on this page.

• Wasserstein & Lazar, “The ASA Statement on p-Values: Context, Process, and Purpose”, The American Statistician 70(2), 2016 — the six principles this page states
• Wasserstein, Schirm & Lazar, “Moving to a World Beyond p < 0.05”, 2019 — the follow-up on thresholds
• Simmons, Nelson & Simonsohn, “False-Positive Psychology” — the paper that quantified optional stopping and analysis flexibility
• Confidence Interval Calculator — the interval that carries the same information plus the effect size
• t-test Calculator — where the statistic on this page usually comes from
• How we calculate — methodology

Method. Tail areas come from this site’s higher-precision distribution engine rather than the one behind the Standard Deviation Calculator. That older engine uses the Abramowitz & Stegun 7.1.26 error function, whose maximum error is about 1.5 × 10⁻⁷ — fine for a confidence interval printed to two decimals, and four orders of magnitude too coarse for a page whose entire answer is a small tail probability. The replacement routes through the regularised incomplete gamma and beta functions, iterated to 1e-15, and is asserted against published table values and against the older implementation to prove the improvement is real rather than claimed. Tail areas are taken from survival functions evaluated directly, never as 1 − CDF: past about |t| = 8 that subtraction underflows to a hard zero, and the first version of this page duly reported p = 0 for the n = 2500 row rather than 4 × 10⁻²³. The sample-size table is computed live from the same engine, not hardcoded. That engine is verified on every change against 155 assertions, including the identity that a two-tailed p is exactly twice the one-tailed p on the same side. The count and the per-case breakdown are published on the formula verification page.

Related calculators

Where this goes next:

More in Math, or browse all calculators.