Cohen’s Kappa Calculator

Identical agreement, kappas two-fold apart

This is the single fact that stops kappa being read as a rescaled percentage agreement, and it is demonstrable in two clicks.

[[40,9],[6,45]] gives κ = 0.6995. The raters agreed on 85% of cases and the marginals were balanced, so chance agreement was 50.08%.

[[80,4],[11,5]] gives κ = 0.3219. The raters agreed on 85% of cases — exactly the same — but both used the first category most of the time, so chance agreement was 77.88% and there was much less room above it.

Same agreement, a factor of 2.17 in kappa. Nothing about the raters’ performance differs between those two tables. What differs is how common the categories are.

And PABAK is exactly 0.70 for both. Prevalence-adjusted bias-adjusted kappa recomputes kappa as if the categories were used equally often; for two categories it is just 2p₀ − 1. Its agreeing on both tables is the cleanest possible statement that the difference was never about the raters.

Neither number is wrong. Kappa correctly says that agreeing 85% of the time is less impressive when 84% of cases are one category. The error is reading kappa as a measure of the raters’ skill alone, and then comparing kappas across studies with different prevalences.

94% agreement and a negative kappa

The third preset pushes the same mechanism to the point where the output looks broken and is not.

[[118,5],[2,0]]: the raters agreed on 118 of 125 cases, 94.4%. And κ = −0.0234.

Chance agreement here is 94.53%. Both raters chose the first category almost always, so two people answering at random with those same rates would have agreed slightly more often than these two did.

So the ratio goes below zero. (0.9440 − 0.9453)/(1 − 0.9453) is a small negative divided by a small positive, and the answer is −0.023.

“Poor agreement” is the wrong reading. The raters agreed constantly. What is poor is the information in that agreement: when almost every case is one category, agreeing carries almost no evidence that either rater can tell the categories apart.

The maximum kappa for those marginals is 0.5614, so even a perfect rearrangement within those row and column totals could not have reached 1. The tool reports that ceiling, and almost nothing else does.

The maximum kappa the marginals permit

Kappa is compared against 1 as though 1 were always attainable. It frequently is not, and the shortfall is computable before you look at how the raters did.

Perfect agreement requires matching marginals. If rater A called 60 cases positive and rater B called 40 positive, at least 20 cases must disagree whatever else happens. The ceiling follows from the totals alone.

So κ = 0.5 means two different things. Against a ceiling of 1 it is half of what was possible; against a ceiling of 0.6 it is 83% of it. The tool prints the ratio.

A low ceiling is itself a finding. It means the two raters are using the categories at systematically different rates — one is more willing to call things positive — which is bias rather than noise, and is fixed by recalibrating rather than by training harder.

κ/κ_max is sometimes reported as a normalised statistic. It has its own critics, because dividing by a data-dependent ceiling makes the result less comparable rather than more. Reporting both numbers avoids choosing.

And it explains why kappa is hard to compare across studies. Two studies with different prevalences and different rater tendencies have different ceilings, so their kappas are not on a common scale even when the task is identical.

Weighting, and when it backfires

With ordered categories, a disagreement between “mild” and “moderate” is not the same as one between “mild” and “severe”. Weighted kappa gives partial credit for near-misses.

Linear weights give credit proportional to the distance, 1 − |i − j|/(k − 1). Quadratic weights fall off with the square, so they are much more forgiving of small disagreements and much harsher on large ones.

On the ordinal preset: 0.6154, 0.7243 and 0.8212. Unweighted, linear and quadratic on one 4×4 table. The choice moves the answer by 0.21.

Quadratic weighting has a second identity. With equally spaced categories it equals the intraclass correlation coefficient computed on the category numbers, which is why it is the default in much of the medical literature and why it can be compared against a reliability coefficient.

The common belief that weighting always raises kappa is conditional. It holds when the disagreements really are concentrated near the diagonal: on 600 ordinally-structured test tables the ordering quadratic ≥ linear ≥ unweighted held every time. On 600 unstructured tables it held about half the time.

So weighting unordered categories can lower it. Giving partial credit for confusing red with green is not a correction; it is an assumption that green sits between red and blue. If the categories have no order, the unweighted kappa is the only defensible one.

The interpretation labels are made up

Almost every kappa is reported next to a word: fair, moderate, substantial, almost perfect. Those words come from one 1977 paper and have no empirical basis.

Landis and Koch proposed them as personal benchmarks, and said so in the paper. They were offered as a convenience, not derived from anything.

The boundaries are arbitrary. 0.61 is “substantial” and 0.60 is “moderate”, which describes a difference that does not exist. Competing scales draw the lines elsewhere, which is what happens when nothing constrains them.

What counts as adequate depends on the use. A kappa of 0.6 between two radiologists on a screening decision is a serious problem; the same figure between two annotators labelling tweets for a sentiment model is fine. The number does not carry that context and the adjective pretends to.

Report what the reader needs instead. The kappa, its interval, the observed agreement, the marginals, and the maximum the marginals permitted. That is five numbers and it says everything the label was standing in for.

And put an interval on it. The tool gives a standard error and a 95% interval. On the extreme preset that interval is enormous, which is the honest summary of a table with two cases in one row — and an adjective would have hidden it.

When kappa is the wrong statistic

Cohen’s kappa answers one specific question: do two raters agree more than chance on a categorical judgement. Several common situations are not that.

More than two raters: Fleiss’ kappa. Cohen’s is defined for exactly two. Fleiss’ generalises to any number, and notably does not reduce to Cohen’s at two — they use different chance models, which is a trap when comparing across papers.

Continuous ratings: the intraclass correlation. Kappa needs categories. For scores on a continuous scale the ICC is the right reliability measure, and quadratically weighted kappa on equally spaced categories is a special case of it.

Agreement against a known truth: sensitivity and specificity. If one of the two columns is a gold standard, the question is accuracy rather than agreement, and the Bayes calculator turns those into the probability that a positive is real.

Systematic disagreement: McNemar’s test. Kappa mixes bias and noise into one number. McNemar tests specifically whether one rater is more likely to call cases positive than the other, which is the question a low ceiling is pointing at.

Ordinal association rather than agreement: Kendall or Spearman. Two raters can rank cases identically while using different thresholds, which gives a high τ and a low kappa. Whether that counts as agreement depends on whether the labels or only the ordering matter.

And raw percentage agreement is not useless. Report it alongside. It is what people actually want to know, and kappa’s job is to say how much of it was luck — not to replace it.

Sources and methodology

References for the coefficient and for the paradox.

• Cohen, “A Coefficient of Agreement for Nominal Scales”, Educational and Psychological Measurement 20(1), 1960
• Feinstein & Cicchetti, “High Agreement but Low Kappa: I. The Problems of Two Paradoxes”, J. Clinical Epidemiology 43(6), 1990
• Byrt, Bishop & Carlin, “Bias, Prevalence and Kappa”, J. Clinical Epidemiology 46(5), 1993 — where PABAK comes from
• Landis & Koch, “The Measurement of Observer Agreement for Categorical Data”, Biometrics 33(1), 1977 — the labels, offered as personal benchmarks
• Kendall Tau Calculator — ordinal association, which is a different question from agreement
• How we calculate — methodology

Method. Kappa is computed from the weight matrix directly, so the unweighted, linear and quadratic versions share one code path and cannot drift apart. The maximum kappa is the value the marginals permit — each category’s agreement capped at the rarer of the two raters’ totals — and PABAK is kappa with balanced marginals, which for two categories reduces to 2p₀ − 1. The suite checks kappa against a longhand 2×2 formula across 400 tables, asserts it never exceeds the maximum its marginals permit, pins the two paradox tables at identical observed agreement with kappas of 0.6995 and 0.3219 and a PABAK of exactly 0.70 for both, and establishes that the quadratic ≥ linear ≥ unweighted ordering holds on all 600 ordinally-structured tables while being close to a coin flip on 600 unstructured ones. That engine is verified on every change against 67 assertions. The count and the per-case breakdown are published on the formula verification page.

Related calculators

Where this goes next:

More in Math, or browse all calculators.