Correlation is not agreement

When a model's output, or a new way of measuring something, is checked against a reference, the first number reported is often a correlation. It is easy to compute and easy to read: 0.98 sounds excellent. I wanted to know exactly what a high correlation guarantees about two measurements, and what it lets through, so I worked through it on paper and then simulated the cases that worried me.

What the coefficient can see

Pearson's coefficient between measurements $a$ and $b$ is

Replace $b$ with $c\,b + d$ for any $c > 0$ and any $d$. The covariance scales by $c$, the standard deviation of $b$ scales by $c$, and the constant $d$ disappears from both. The coefficient doesn't change. So a correlation is blind, by construction, to any constant offset and to any proportional error. It answers "do they move together?", which is a different question from "are they the same?".

That is the argument Bland and Altman (1986)¹ made against using correlation to compare two methods of measurement. Their alternative is to look directly at the differences $b - a$: their mean is the systematic bias, and the mean plus and minus 1.96 standard deviations gives the range within which 95% of individual disagreements fall, the limits of agreement. Plotting the differences against the averages $(a + b)/2$ also shows whether the disagreement grows with the size of the quantity.

Three second methods

I simulated 200 items with true values drawn from a normal distribution with mean 50 and standard deviation 10. Method A measures each with noise of standard deviation 1.5. Method B does one of three things: the same as A with its own noise; the same plus a constant offset of 8; or 1.3 times the truth minus 12, which is a proportional error that roughly cancels near the middle of the range.

The offset method has the same correlation as the honest one. The proportional method has the highest correlation of all three, because stretching the scale stretches the true spread that the coefficient divides by. A table of correlations would rank it first.

The intraclass correlation coefficient is the usual next step, and it comes in forms that answer different questions. Shrout and Fleiss (1979)² and McGraw and Wong (1996)³ separate the consistency form, which ignores systematic differences between methods, from the absolute-agreement form, which counts them as error. The consistency ICC behaved exactly like Pearson's r and stayed at 0.98 under the offset. The absolute-agreement ICC fell to 0.73. Reporting "ICC" without saying which one leaves open the very question it was meant to answer.

The Bland–Altman view showed both problems at once. The offset appears as a mean difference of 8 with limits that exclude zero. The proportional error appears as a slope of 0.27 in the differences against the averages, so the disagreement is small in the middle of the range and large at both ends.

A curve with the right shape

The same blindness applies to a predicted signal compared with a true one. I made a true curve, $30\sin t + 10\sin 3t$, and a prediction that is 0.8 times the curve plus 15, with a little noise. The correlation between them over the whole curve is 0.994. The root-mean-square error is 15.9, while the true curve's average magnitude is 21. The prediction has the shape almost exactly and the level badly wrong, and a correlation can only report the first. A high correlation is genuine evidence that a model tracks how a quantity changes. It says nothing about whether the values can be used as measurements.

Three points are not enough to correlate

There is a second, quieter way a correlation misleads: when it is computed from very few points. I drew pairs from a population whose true correlation is 0.8 and computed the coefficient from 3, 10 and 100 pairs, many thousands of times each.

From three points the coefficient can land almost anywhere. One estimate in ten came out negative for a population whose correlation is strongly positive, and the median, 0.91, is higher than the truth. With three points a line fits almost perfectly by chance, so a small-sample correlation is both noisy and inflated. Correlating each series over all its points, and then summarising those, is far steadier.

RMSE and MAE are different questions too

Once correlation is replaced by an error measure, a similar choice comes up between the root-mean-square error and the mean absolute error. For normally distributed errors their ratio is fixed: $\sqrt{\pi/2} \approx 1.25$. I checked that, and it came out at 1.25. Then I made 5% of the errors ten times larger.

The tail more than doubled the RMSE, raised the MAE by half and barely moved the median. None of them is wrong; they weigh rare large errors differently. An RMSE/MAE ratio well above 1.25 is itself a cheap signal that the errors have a tail. In that case the median says what a typical case looks like, and the RMSE says what an acceptance threshold has to survive.

What to report when comparing against a reference

These are the numbers I would put in front of a reader when a model or method is compared against a reference:

• the mean difference and the 95% limits of agreement, in the quantity's own units;
• the slope of the differences against the averages, or the Bland–Altman plot itself;
• the absolute-agreement ICC, named as such, if an ICC is wanted at all;
• correlation only as a statement about tracking, computed over whole series, never over a handful of points;
• RMSE, MAE and the median absolute error together, so that a tail can't hide.

The error bars on any of these numbers depend on how many independent items there are, which can be far fewer than the count suggests, in a hundred thousand correlated samples can count as five hundred. Another single number that hides what matters is average precision, in average precision scores thresholds you will never use.

What I still don't know

All of these simulations use independent items and normal noise apart from the one tail. Repeated measurements of the same items, or errors that drift over time, need the repeated-measures versions of these methods, which I haven't worked through. I also haven't looked at how the limits of agreement should be set when the noise itself grows with the quantity. The usual advice is to work on a log scale, and I'd like to see how much that changes the picture for the proportional case above.

References

1. Statistical Methods for Assessing Agreement Between Two Methods of Clinical Measurement — J. Martin Bland and Douglas G. Altman, The Lancet 1986
2. Intraclass correlations: Uses in assessing rater reliability — Patrick E. Shrout and Joseph L. Fleiss, Psychological Bulletin 1979
3. Forming inferences about some intraclass correlation coefficients — Kenneth O. McGraw and S. P. Wong, Psychological Methods 1996