t-test Calculator

Three tests share the name, and the choice comes first

Picking the wrong design is a bigger error than any option inside a design, and it is decided by how the data was collected rather than by what you want to find out.

One-sample compares a single mean against a fixed value you specify. Is the average fill weight 500 g? Does this class differ from the national average? There is one group and one number to test it against.

Two-sample compares two independent groups: different people, different items, nothing linking a particular member of one to a particular member of the other. Treatment against control, variant A against variant B.

Paired is for measurements that come in linked couples — the same subject before and after, the same patient on each of two drugs, matched siblings. Here you compute the difference within each pair and run a one-sample test on those differences against zero.

The consequence of getting it wrong is not subtle. Running a paired study as a two-sample test throws away the pairing, which was the entire point: pairing removes between-subject variation, and that variation is usually much larger than the effect. A paired analysis can be dramatically more powerful on identical data, so the mistake costs you the study.

The reverse error — pairing genuinely independent groups — is normally impossible to do accidentally, because there is no basis on which to match the observations.

Why Welch is the default here

The two-sample t-test comes in two versions, and the one taught first is the one you should usually not use.

Student’s version assumes the two populations have the same variance. It pools them into a single estimate and gets clean integer degrees of freedom, n₁ + n₂ − 2. That assumption is what buys the tidiness.

Welch’s version makes no such assumption. It keeps the two variances separate and computes fractional degrees of freedom from the Welch–Satterthwaite formula — which is why the tool reports something like 29.708 rather than a round number.

The case for Welch as the default rests on three things.

The assumption is rarely checked and often false. Groups that differ in mean frequently differ in spread too — a treatment that raises an average often changes variability with it.

The failure is not graceful. When variances differ and group sizes are unequal, Student’s test does not become slightly conservative — its actual error rate departs from the nominal one, and in the direction that reports significance too readily when the smaller group has the larger variance. Since unequal group sizes are the normal condition of real data, that is not an edge case.

It costs almost nothing when the assumption holds. With equal variances and equal n the two are algebraically identical — this page’s suite asserts exactly that. Otherwise Welch gives up a small amount of power for a test that is right either way.

This is why R’s t.test() has defaulted to Welch for years. Student’s version remains on screen because coursework usually asks for it.

Do not test for equal variances first

A procedure that looks careful and is not: run an F-test or Levene’s test for equal variances, then pick the t-test version based on the result.

It inflates the error rate. You have made the choice of test conditional on the data, so the final p-value no longer means what it claims. The two-stage procedure has a true error rate higher than the nominal one, and the amount depends on things you cannot see.

The pre-test is underpowered where it matters. With small samples — exactly when the variance assumption matters most — a test for equal variances has little power to detect a real difference. It will usually fail to reject, sending you to Student’s version precisely when it is least safe.

And it is unnecessary. Welch is valid whether the variances are equal or not, so there is nothing to decide. Simulation studies through the 1990s and 2000s converged on the same recommendation: use Welch unconditionally.

The tool follows that. It reports the variance ratio and flags a large mismatch as context — a factor of four is worth knowing about — but it does not switch tests on the basis of it, and neither should you.

What a t-test actually requires

Three conditions, and they are not equally important. It is worth knowing which one to worry about.

Independence, which matters most. Observations must not influence one another. Repeated measurements on the same subject, pupils within one classroom, or readings taken in sequence from a drifting instrument all violate it, and the test has no defence — it will report confidently wrong answers. This is the assumption to check first and the one no sample size fixes.

Normality, which matters least. The t-test assumes the sampling distribution of the mean is normal, not that the data is. The central limit theorem does the work, so for mildly skewed data a few dozen observations is ample. Below about 15 per group with visibly skewed data it starts to matter, and heavy tails or strong skew are better handled by a rank-based test such as Mann-Whitney, or by a bootstrap.

Equal variances, only for Student’s version. Welch removes this requirement entirely, which is the whole argument of the section above.

One more that is rarely listed: the data has to be measured on a scale where a mean is meaningful. Running a t-test on ranks, or on a five-point rating treated as though the gaps between points were equal, produces a number without producing a result.

The p-value is half the answer

A t-test tells you whether a difference is detectable. It cannot tell you whether it is worth anything, which is why the tool shows Cohen’s d beside the p-value rather than below it.

Cohen’s d is the difference in standard deviations. A d of 0.5 means the two group means are half a standard deviation apart, whatever the units were. The conventional labels are 0.2 small, 0.5 medium, 0.8 large — and Cohen himself described them as a last resort for fields with no better yardstick.

The two quantities behave completely differently as data accumulates. d does not change systematically with sample size; p does. Collect enough observations and any non-zero difference becomes significant, which the p-value Calculator demonstrates directly. d stays where it is.

So the two failure modes are opposite and both common. A very large study reports p below 0.001 for a d of 0.05 — real, detectable, and too small to act on. A small study reports p = 0.2 for a d of 0.8 — a large effect it lacked the power to establish.

The honest report gives the difference in its own units, a confidence interval for it, the effect size, and the p-value. In that order.

What a non-significant result does not establish

“No significant difference” gets written up as “no difference” constantly, and the two are not the same claim.

Failing to reject is not accepting. The test asks whether the data is inconsistent with the null. Finding it is not inconsistent leaves you with the null and every small effect the study was too small to detect, all still on the table.

Look at the confidence interval instead. It separates two situations a p-value cannot. A narrow interval tightly around zero is genuine evidence the effect is small. A wide interval spanning zero says the study learned very little. Both give p above 0.05.

If you need to show equivalence, test for it. There is a proper method — equivalence testing, usually two one-sided tests — which asks whether the effect is smaller than some threshold you specify in advance. It requires you to say what “no meaningful difference” means numerically, which is the useful discipline. An ordinary t-test cannot deliver that conclusion however large the p-value.

The wording that survives scrutiny: “we did not detect a difference; the 95% interval for the difference runs from −1.2 to 3.4, so effects up to about 3 points remain compatible with our data.”

Why the degrees of freedom come out fractional

A Welch t-test reporting df = 29.708 looks like an error to anyone expecting an integer. It is correct, and the reason is worth understanding.

Degrees of freedom in a t-test index which member of the t family to use as the null distribution. Nothing requires that index to be a whole number — the family is defined for any positive real value, and the distributions in between are perfectly well-behaved.

Student’s version gets an integer because pooling two variance estimates into one adds their degrees of freedom exactly: (n₁−1) + (n₂−1).

Welch keeps them separate, so it is combining two estimates with different precisions. The Welch–Satterthwaite formula finds the single t distribution that best approximates the resulting sampling distribution, and there is no reason for the answer to land on an integer.

It is always between the smaller group’s df and the pooled df. That bracket is a good sanity check. When the variances are equal and the samples are the same size, it lands exactly on the pooled value — which is why the two versions coincide there, and what this page’s suite asserts.

Printed tables have no row for 29.708, which is the historical reason Welch was unpopular. Rounding down is conservative and was the old workaround; there is no longer any need for it.

Sources and methodology

The papers behind the default, and the reference implementation.

• Welch, “The generalization of Student’s problem when several different population variances are involved”, Biometrika 34, 1947 — the original
• Ruxton and later simulation work on unequal variances — the basis for recommending Welch unconditionally rather than after a pre-test
• R documentation for t.test() — which defaults to Welch, as this page does
• p-value Calculator — what the p from this test can and cannot support
• Confidence Interval Calculator — the interval that makes a non-significant result readable
• How we calculate — methodology

Method. Welch is the default and Student’s pooled version is computed alongside it on the same data, so the comparison is on the page rather than described. The Welch–Satterthwaite degrees of freedom are the easiest quantity here to get subtly wrong, so the suite checks them against a hand computation and against the boundary case where Welch must collapse exactly onto Student — equal variances with equal n, where the two are algebraically identical. A df formula wrong in general would almost certainly still fail that. The suite also asserts that the paired test is exactly a one-sample test on the differences, that a two-tailed p is exactly twice the one-tailed p on the same side, and that the reject-at-α verdict agrees with the critical-value comparison in all three hundred generated cases. Tail probabilities come from survival functions rather than 1 − CDF, so a large t does not underflow to p = 0. That engine is verified on every change against 155 assertions. The count and the per-case breakdown are published on the formula verification page.

Related calculators

Where this goes next:

More in Math, or browse all calculators.