Intraclass Correlation Calculator

The six forms are six different questions

Shrout and Fleiss named them in 1979 and the naming has been ignored ever since. The forms are not refinements of one another; they answer different questions and can differ by more than the whole usable range.

The first number is the design. 1 means each subject was rated by a different, randomly chosen rater. 2 means the same raters rated everyone and are a sample of a wider pool. 3 means the same raters rated everyone and are the only raters of interest.

The second number is what you will actually use. (…,1) is the reliability of a single rating. (…,k) is the reliability of the mean of k raters, which is always higher, and is the right one only if your real procedure averages raters.

The gap between forms 2 and 3 is the whole question of rater bias. Form 2 counts a rater who is consistently 14 points low as disagreement. Form 3 does not, because it removes the rater effect first.

On the tool’s second preset that single choice moves the answer from 0.1277 to 0.9852. Rater 2 runs about 14 points low and rater 3 about 11 high, consistently across all eight subjects. Every rater ranks the subjects almost identically; none of them agree on the value.

Choosing the form from the study, not from the result

The form is determined by how the data was collected and how the measurement will be used. It is not a modelling preference, and it is certainly not a choice to be made after seeing the values.

Did every subject get the same raters? If not — different clinicians saw different patients — the design is one-way and only form 1 applies. Forms 2 and 3 are not computable from that design even though the arithmetic will produce a number.

Do you care about absolute values or only ranking? If a measurement will be compared against a threshold, systematic bias matters and you want absolute agreement — form 2. If only the ordering matters, consistency — form 3 — is the honest choice.

Will the deployed measurement be one rating or an average? If clinical practice uses a single rater’s score, report (…,1). Reporting (…,k) for a procedure that uses one rater overstates the reliability of what people will actually do.

Reporting the highest of the six is the failure mode this page is about. It is usually ICC(3,k), which is the most permissive combination available, and it is almost never the one the study design supports.

A negative ICC, and why it is not clamped

The ICC is a ratio of variance components, and a variance component estimated by subtraction can come out below zero.

It means the within-subject disagreement exceeds the between-subject variation. Raters disagree about a given subject more than subjects differ from each other — which is worse than no reliability, not equal to it.

Software that returns 0 instead is hiding the diagnostic. A true value of 0 and an estimate of −0.41 mean different things about a dataset, and the second usually means something is wrong upstream.

Small samples produce negative estimates by chance even when the true ICC is positive, which is a reason for a wide confidence interval rather than a reason to round up.

The tool reports the value the arithmetic gives. The third preset is negative on every form, and that is the correct description of raters who disagree more than chance predicts.

ICC, kappa and Pearson answer different things

All three get called “agreement” and only one of them is about agreement in the sense most people mean.

Pearson’s r is blind to systematic bias entirely. A rater who scores every subject exactly 14 points below another correlates with them at r = 1. That is the same blindness ICC(3,1) has, which is why the consistency form and a correlation often land close together.

Pearson also only handles two raters. With three or more you would be averaging pairwise correlations, which is not a defined reliability coefficient. See the correlation coefficient calculator for what it does measure.

For categorical ratings the ICC is the wrong tool. Cohen’s kappa handles nominal categories and corrects for chance agreement, which a variance ratio cannot do.

And for consistency across ITEMS rather than raters, alpha is the counterpart. Cronbach’s alpha is arithmetically close to ICC(3,k) with items in place of raters — the same mean squares, a different substantive question.

The 0.75 and 0.90 thresholds are conventions, not findings

Koo and Li’s bands — below 0.5 poor, 0.5 to 0.75 moderate, 0.75 to 0.90 good, above 0.90 excellent — are widely quoted and were offered as rules of thumb rather than derived cutoffs.

The required reliability depends on the decision. A measurement used to screen a population tolerates far less than one used to decide an individual’s treatment, and no single band covers both.

The confidence interval usually straddles a band boundary. With 20 subjects and 2 raters the interval around an ICC of 0.80 is wide enough to include both “moderate” and “excellent”, which makes the label less informative than the number.

An ICC also depends on the sample’s heterogeneity. The same instrument, with identical measurement error, reports a higher ICC on a more varied sample — because the between-subject variance in the numerator is larger. It is not a property of the instrument alone.

Which makes the standard error of measurement the more portable number. It is in the original units and does not move with the sample’s spread, and reporting it alongside the ICC costs one line.

Reporting an intraclass correlation

Four elements, and the first is the one most commonly omitted.

Name the form. “ICC(2,1) = 0.61” is a result. “ICC = 0.61” is not, because the same data supports five other numbers.

Give the number of subjects and the number of raters. They determine the degrees of freedom and therefore how much the estimate can be trusted.

Give a confidence interval. ICCs from small samples are unstable, and the point estimate alone implies a precision the design does not have.

And say whether the deployed measurement is a single rating or an average. It decides between (…,1) and (…,k), and reporting the wrong one overstates what the procedure achieves in practice.

Sources and methodology

References for the ICC forms and their interpretation.

• Shrout & Fleiss, “Intraclass Correlations: Uses in Assessing Rater Reliability”, Psychological Bulletin 86(2), 1979 — where the six forms are named
• Koo & Li, “A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research”, Journal of Chiropractic Medicine 15(2), 2016
• McGraw & Wong, “Forming Inferences About Some Intraclass Correlation Coefficients”, Psychological Methods 1(1), 1996
• Cohen’s Kappa Calculator — the categorical-rating counterpart
• Cronbach’s Alpha Calculator — the same mean squares applied to items instead of raters
• How we calculate — methodology

Method. The four mean squares are computed from the two-way ANOVA decomposition of the subject-by-rater matrix, and all six forms are derived from those same four numbers — so the forms can never disagree about the data underneath them, only about what counts as error. The spread between single-measure forms is computed rather than described, which is what makes the central claim checkable: the preset matrix gives ICC(1,1) = 0.127727 and ICC(3,1) = 0.985176, a spread of 0.857450, with F = 200.3810 and p = 6.09e-13. Negative values are returned as computed and never clamped to zero. Ragged rows, a single subject and a single rater all return no result rather than a number. That engine is verified on every change against 69 assertions. The count and the per-case breakdown are published on the formula verification page.

Related calculators

Where this goes next:

More in Math, or browse all calculators.