Correlation Field Guide

A coefficient is a summary, and every summary throws something away. Which correlation measure sees which kind of dependence, and what to actually do once two features turn out to carry the same information.

Every figure on this page is computed in your browser from a seeded generator: nothing is pre-computed, nothing is sent anywhere.

Contents

Part one

Measuring dependence

Every coefficient below answers a slightly different question. Pearson asks how well a straight line fits. Spearman asks whether the ordering agrees. Distance correlation asks whether the two variables are independent at all. Give them the same data and they will disagree — that disagreement is itself the diagnostic.

Five measures, one sample

Pick a shape. All five coefficients are recomputed on the same 200 points, so you can watch them come apart. Switch to ranks to see the plot exactly as Spearman and Kendall see it — the values are replaced by their positions in the ordering.

Shape

Noise

0.30

View

Synthetic sample, n = 200, drawn from a seeded generator so every reading is reproducible. Mutual information uses 7 equal-frequency bins per axis, normalised by √H(X)H(Y). The two unsigned measures are biased upward at finite n: on genuinely independent data at this sample size, distance correlation still averages 0.12 and normalised MI 0.05. The baselines quoted on the tiles are the 95th percentiles under independence, measured by simulation — a reading below them means nothing found, not weak dependence.

Choosing by the type of the two variables

Most of the choice is settled before you look at the data at all: the measurement scales of the two variables rule out most of the options.

Variable A	Variable B	Default choice	Reach for instead when
Continuous	Continuous	Pearson r	Heavy tails, outliers, or a curved-but-monotonic relation → Spearman
Ordinal	Ordinal	Kendall τ_b	Few ties and n comfortably above ~50 → Spearman reads more familiarly
Continuous	Binary	Point-biserial r_pb	The binary split is very unbalanced → report the group means and an effect size
Binary	Binary	Phi φ	You want an interpretable ratio → odds ratio or lift
Nominal (k levels)	Nominal (m levels)	Cramér's V	Many rare levels — V is biased upward → use the bias-corrected variant
Continuous	Nominal (k levels)	Correlation ratio η²	Strongly non-normal groups → ε² from a Kruskal–Wallis test
Anything	Anything	Distance correlation	You want to see where the dependence lives → the binned joint table behind mutual information
Two variables, with a third to hold fixed		Partial correlation r_xy·z	You want the effect of z removed from one side only → semi-partial

The coefficients

Range, assumptions, and the failure mode that actually bites in practice.

Pearson correlation

product-moment

r_xy = Σ_i(x_i − x)(y_i − y) √Σ_i(x_i − x)² · Σ_i(y_i − y)²

sees: linear range −1 … 1 interval scale

Reads asCovariance divided by the two standard deviations. r² is the share of variance in y a straight line in x explains.

AssumesA linear relation and finite variance. Not normality or symmetric errors — those are only needed for the p-value.

Use whenBoth variables are continuous and you genuinely care about a straight-line fit — linear models, PCA, anything built on a covariance matrix.

Watch outA single high-leverage point can drive r from 0 to 0.8. Curvature makes it read zero on a perfect relationship. Range restriction (filtering the sample) attenuates it.

Spearman correlation

rank-order

ρ = 1 − 6 Σ_i d_i² n(n² − 1)

sees: monotonic range −1 … 1 ordinal scale

Reads asPearson computed on the ranks. The shortcut above holds only when there are no ties; with ties, rank first and run Pearson.

AssumesOnly that the relation is monotonic and the data can be ordered. d_i is the rank difference for observation i.

Use whenSkewed variables, outliers, ordinal scales, or a relation you expect to be monotonic but not straight — prices, counts, durations, survey items.

Watch outStill blind to non-monotonic shapes (a U keeps reading zero). ρ² is not a variance share. Ranking discards the magnitude of the gaps, which is sometimes the thing you cared about.

Kendall tau-b

concordance

τ_b = C − D √(n₀ − n₁)(n₀ − n₂)

sees: monotonic range −1 … 1 handles ties

Reads asA probability difference: P(concordant) − P(discordant) over all pairs. C and D count agreeing and disagreeing pairs; n₀ is all pairs, n₁ and n₂ the pairs tied on x and on y.

AssumesNothing beyond an ordering. It is the most distribution-free of the three.

Use whenSmall samples, many ties, or when you want a coefficient with a direct probabilistic meaning you can state to a stakeholder.

Watch outNumerically smaller than Spearman on the same data — roughly τ ≈ ⅔ρ for well-behaved data — so never compare the two across reports. Naive computation is O(n²); Knight's algorithm makes it O(n log n) and is what scipy.stats.kendalltau uses.

r_pb

Point-biserial

continuous ↔ binary

r_pb = M₁ − M₀ s_n √ n₁n₀n²

sees: mean shift range −1 … 1 = Pearson

Reads asThe standardised gap between the two group means, scaled by how balanced the split is. It is algebraically identical to Pearson with the binary coded 0/1 — there is no separate estimator to implement.

AssumesA genuine dichotomy. If the binary was made by cutting a continuous variable, biserial correlation is the right relative.

Use whenScreening continuous features against a binary target, or a binary flag against a continuous outcome.

Watch outBounded away from ±1 by the class balance: a 95/5 split caps r_pb well below 1 even when the two groups do not overlap at all — the cap comes from matching a two-point marginal against a spread-out continuous one. Compare across features with similar balance only.

Cramér's V

nominal ↔ nominal

V = √ χ² n · min(k − 1, m − 1)

sees: any association range 0 … 1 no sign

Reads asThe χ² statistic of the contingency table, normalised so it cannot exceed 1. For a 2×2 table it collapses to |φ|.

AssumesEnough observations per cell for χ² to be meaningful — the usual rule of thumb is an expected count of 5.

Use whenBoth variables are categories with no order: country, device, product line, error code.

Watch outStrongly biased upward when either variable has many levels relative to n — a high-cardinality ID column will look "associated" with everything. Use the bias-corrected version (Bergsma 2013: replace φ² = χ²/n by max(0, φ² − (k−1)(m−1)/(n−1)) and shrink k and m the same way), and never rank features by raw V across different cardinalities.

η²

Correlation ratio

continuous ↔ nominal

η² = Σ_g n_g(y_g − y)² Σ_i(y_i − y)²

sees: group separation range 0 … 1 = ANOVA

Reads asThe share of the variance in y explained by knowing which group a row belongs to. Between-group sum of squares over total.

AssumesNothing about the ordering of the groups — that is the point. It is the one-way ANOVA effect size.

Use whenAsking whether a categorical feature carries information about a numeric target, or whether a numeric feature is redundant with a category.

Watch outAlso inflated by high cardinality — with one group per row, η² is exactly 1. Cross-validate, or use the adjusted ω².

dCor

Distance correlation

Székely–Rizzo

dCor²(X,Y) = dCov²(X,Y) √dVar²(X) dVar²(Y)

sees: any dependence range 0 … 1 no sign

Reads asPearson correlation between the doubly-centred pairwise distance matrices of X and Y. Its defining property: dCor = 0 if and only if the two are independent.

AssumesFinite first moments. Works for vectors of different dimension, which pairwise r cannot do.

Use whenYou suspect a relationship the linear measures are missing, or you need a screening statistic that will not silently return zero on a real dependence.

Watch outO(n²) in memory and time — subsample above ~10k rows. It gives no direction and no shape: a high value tells you to go and plot the pair, nothing more.

Mutual information

information gain

I(X;Y) = Σ_xΣ_y p(x,y) log p(x,y) p(x)p(y)

sees: any dependence range 0 … ∞ estimator-dependent

Reads asHow many nats of uncertainty about Y disappear once you know X. Zero exactly when they are independent. Normalise by the entropies to get a 0–1 number you can compare across pairs.

AssumesNothing about the form — but for continuous variables you must choose a density estimator, and the choice moves the number.

Use whenMixed variable types in one screen, or when you want a single measure that works for categorical and numeric alike.

Watch outPositively biased at small n — independent variables score above zero, and the bias grows with the number of bins. Fix the binning across all pairs before you rank features, or use a k-nearest-neighbour estimator.

r_xy·z

Partial correlation

controlling for z

r_xy·z = r_xy − r_xzr_yz √(1 − r_xz²)(1 − r_yz²)

sees: residual linear range −1 … 1 multivariate

Reads asThe correlation between the residuals of x and y after each has been regressed on z. In matrix form, the negated, normalised inverse covariance — the precision matrix.

AssumesThe control relations are linear. For a general control, regress out with a flexible model and correlate the residuals.

Use whenTwo features look correlated only because both track a third thing — time, size, region. This separates shared drivers from direct redundancy.

Watch outConditioning on a collider creates correlation that was not there. Which variables you control for is a causal claim, not a statistical convenience.

Phi coefficient

binary ↔ binary

φ = ad − bc √(a+b)(c+d)(a+c)(b+d)

sees: 2×2 association range −1 … 1 = Pearson

Reads asPearson on two 0/1 variables, written in terms of the four cell counts a b c d of the 2×2 table.

AssumesTwo genuine dichotomies.

Use whenChecking whether two binary flags — two rules, two labels, two one-hot columns — are carrying the same signal.

Watch outLike point-biserial, its maximum is capped by the marginals: two rare events with different base rates — say 1% and 10% — can have one imply the other and still score φ ≈ 0.3. Report the odds ratio alongside it.

Four things a coefficient will not tell you

The shape

Anscombe's quartet and the Datasaurus dozen are the standard demonstrations: wildly different point clouds sharing the same means, variances and r to two decimals. Plot the pair before you act on the number.

The direction of causation

A coefficient is symmetric — r_xy = r_yx — so it cannot contain directional information even in principle. Anything causal comes from the design, not from the statistic.

Whether the sample is one population

Simpson's paradox: a relation that is negative inside every subgroup can be positive overall. Try the two clusters shape in the sandbox. Always ask what the rows are.

How much you should believe it

With n = 20, the 95% interval around an r of 0.5 spans roughly 0.07 to 0.77. Bootstrap the coefficient before you build a decision on the third decimal.

Part two

Handling correlated features

Collinearity is not a data-quality defect to be scrubbed. It is a statement about what your data can and cannot identify. Before reaching for a fix, be precise about which of the two things it actually breaks — because it reliably breaks one and reliably leaves the other alone.

What it breaks, and what it does not

Breaks: the individual coefficients

When two features move together, many different weightings of them fit the data almost equally well. The standard errors inflate, signs flip between refits, and "feature x is worth 0.4" stops being a defensible sentence.

Leaves alone: the predictions

The fitted surface is stable even when the coefficients are not, so held-out error barely moves. If the model only ever gets asked for y_hat, collinearity may cost you nothing at all.

Breaks: attribution and importance

Tree ensembles split arbitrarily among correlated features, splitting the credit and making each look weak. Permutation importance understates all of them, because the information survives in the twin.

Leaves alone: identifiable combinations

Even at r = 0.99 the sum of two coefficients is estimated precisely; only the split between them is unstable. That is often the quantity you wanted anyway.

What collinearity does to a coefficient

A textbook setup, simulated live: y = x₁ + x₂ + ε, so both true coefficients are exactly 1. Each dot is one refit of the same model on a fresh sample of 60 rows. Raise the correlation between x₁ and x₂ and watch the cloud stretch along the line where the sum stays at 2 — the data pins down the total and says almost nothing about the split.

corr(x₁, x₂)

0.90

Ridge λ

0.00

one refit of the model true value (1, 1)

Dashed line: all coefficient pairs summing to 2. The cloud collapses onto it as the correlation rises.

240 independent samples of n = 60, σ = 1. Test error is measured on 200 fresh rows per fit. The ridge penalty adds λ · n to the diagonal of X′X on the centred predictors, which have unit variance by construction, so λ = 0 is ordinary least squares and λ is on the per-observation scale.

Detecting it — pairwise r is not enough

A dozen features can each be only weakly correlated in pairs and still be almost perfectly linearly dependent together. Pairwise correlation cannot see that; the diagnostics below can.

Diagnostic	Definition	Rough threshold	Catches
Pairwise \|r\|	Correlation matrix of the features	> 0.8 worth a look	Two-way redundancy only. Cheap, and the right first pass.
VIF	VIF_j = 11 − R_j² — regress feature j on all the others	> 5 suspicious, > 10 severe	Multi-way dependence. √VIF is the factor by which the standard error is inflated.
Condition index	κ = √λ_max / λ_min of X′X on standardised columns	10–30 moderate, > 30 severe	The overall conditioning in one number. Standardise first, or it measures your units; on the correlation matrix that is already done.
Eigenvalue spectrum	Eigenvalues of the correlation matrix	Any near zero	Near-zero eigenvalues, whose eigenvectors name the offending combination directly.
Refit variance	Coefficient spread across bootstrap resamples or CV folds	Sign flips	The problem as it will actually reach you. Works for any model, no assumptions.

Pruning a correlation matrix

Eleven features of a property dataset, generated from three latent factors — size, location and age — plus per-feature noise, which is why the block structure is real rather than decorative. Two pairs are exact duplicates by construction: a unit conversion, and a variable stored both as a year and as an age. Move the threshold to see the greedy pruner work.

Drop above |r|

0.85

Redundancy groups at this threshold

Result

The rule is the standard greedy one (caret::findCorrelation): take the most correlated surviving pair, and drop whichever of the two has the higher average correlation with everything else still in the set. Every number in the matrix is printed, so the colour is a reading aid rather than the only encoding. The slider goes down to 0.50 so you can watch the rule over-prune; in practice anything below ~0.85 is throwing away real information.

Eight treatments

Ordered from least to most invasive. The first one is not a joke — it is the correct answer more often than the rest combined.

Do nothing

Cost: none

Leave it alone

If the model exists to produce predictions and nobody will read its coefficients, correlated features are not a problem to solve. Held-out error is essentially unaffected — the second lab shows the test RMSE holding flat while the coefficient spread grows by an order of magnitude.

Dropping features always throws away some information. Doing it to satisfy an assumption your use case never depended on is a pure loss.

Use when Pure prediction; regularised or tree-based models; you have no attribution story to tell. Do not use when Anyone will quote a coefficient, a feature importance, or a "driver" from this model.

Select

Cost: information loss

Drop one of each redundant pair

The greedy threshold rule in the lab above. Simple, reproducible, and it keeps the surviving features exactly as interpretable as they were.

Choose the survivor on grounds the correlation matrix does not know about: which is cheaper to collect, which is measured more reliably, which is available at prediction time, which a domain expert can explain. Picking the one with the higher correlation to the target is the obvious rule and the weakest one — it overfits the training sample and ignores interactions.

Use when The redundancy is genuine duplication — unit conversions, near-synonyms, a rolling mean beside its own input. Do not use when The two features differ in the tails or on a subpopulation. A global r of 0.95 can hide the segment you care about.

Group

Cost: a tuning choice

Cluster the features, keep one per cluster

Run hierarchical clustering on the distance 1 − |r| (Spearman if the features are skewed), cut the dendrogram at a chosen height, and take one representative — or the cluster mean — per group.

This scales where pairwise pruning does not, and it produces a stable, reviewable artefact: a named list of redundancy groups that you can hand to a domain expert. FeatureAgglomeration in scikit-learn and VarClusHi in Python both implement this directly.

Use when Hundreds of features with block structure — sensor arrays, financial ratios, aggregated behavioural counts. Do not use when The clusters have no meaning anyone can name; you have then just made an opaque choice look principled.

Engineer

Cost: domain effort

Combine them into the quantity you meant

Height and weight are collinear; BMI is not. Total spend and transaction count are collinear; average basket size is not. Price and area are collinear; price per square metre is not.

This is the same insight as the second lab, from the other end: the sum was always the identifiable quantity, so estimate the sum. A ratio, a difference or an index makes the stable direction explicit and drops the unstable one on purpose.

Use when A meaningful combination exists. This is the best available answer whenever it does. Do not use when You are inventing a combination to satisfy a diagnostic rather than to express something real.

Regularise

Cost: bias, a hyperparameter

Ridge, lasso, elastic net

Ridge is the direct treatment: the L2 penalty makes the design matrix invertible again and shrinks correlated features together, splitting the shared effect between them instead of letting the fit pick a random split. Both stay in the model. Move the ridge slider in the second lab and watch the cloud contract.

Lasso does the opposite: among a correlated group it keeps roughly one and zeroes the rest — but which one changes from resample to resample. It is a sparsity tool, not a collinearity tool, and its selections should never be read as importances. Elastic net is the compromise: the L2 part restores the grouping effect so correlated features enter or leave together.

Use when Linear or generalised linear models where you want stable coefficients and all features retained. Do not use when You forgot to standardise. Penalties are scale-dependent; fit the scaler on the training fold only.

Project

Cost: interpretability

PCA, PLS and friends

PCA replaces the features with orthogonal components, so collinearity is gone by construction. The price is total: the model no longer has features with names, and every downstream explanation has to be given in terms of loadings.

PCA is also unsupervised — it ranks directions by variance in X, with no reference to y, so a low-variance direction that happens to predict the target can be discarded. Partial least squares fixes exactly that by extracting components that covary with y, and usually needs fewer of them.

Use when Wide, dense, homogeneous blocks — spectra, images, embeddings — where individual columns were never interpretable anyway. Do not use when The deliverable is an explanation. Also never fit the projection on the full dataset before splitting.

Residualise

Cost: an arbitrary ordering

Regress one out of the other

Keep x₁ as it is and replace x₂ with its residual after regressing it on x₁. The two are now exactly uncorrelated and both stay in the model.

The catch is in the meaning: the second feature is now "the part of x₂ that x₁ does not explain", and all of the shared effect has been assigned to x₁ by your choice of ordering. That is a defensible move when there is a real hierarchy — a baseline and an adjustment, a control and a treatment — and pure sleight of hand otherwise.

Use when There is a principled order: a known driver plus a residual effect you want to isolate. Do not use when The ordering is arbitrary — you have hidden a modelling decision inside a preprocessing step.

Attribute

Cost: compute

Fix the importance measure instead of the features

If collinearity is only hurting your explanations, treat the explanation. Permuting one of two correlated features leaves the information intact in its twin, so both look unimportant — and the permutation also creates rows that could not exist, which pushes the model off its training distribution.

Permute the whole correlated group at once and report one number for the group; or use conditional permutation, which resamples within strata of the correlated features. The honest output is "these three features jointly carry 12% of the signal", not three separate small numbers.

Use when The model is fine and only the attribution is misleading — the common case with gradient boosting. Do not use when You need per-feature numbers for a decision. Then go back and fix the feature set.

Picking a treatment by what you owe the reader

Your goal	First move	Fallback	Skip
Predictive accuracy, black box	Nothing	Ridge or a tree ensemble	Truncated PCA — you would lose signal for no gain
Interpretable coefficients	Combine into a meaningful quantity	Ridge, then report the group effect	Lasso — its selection among a correlated group is not stable
Causal or policy inference	Fix the design — more variation, or a different identification strategy	Report what is identified: the sum, the contrast	Any automatic pruner. Dropping a confounder to reduce VIF creates bias
Feature importance for a stakeholder	Grouped permutation importance	Cluster first, then report per cluster	Raw gain-based importances from a boosted tree
Cutting cost — fewer columns to collect	Cluster, keep the cheapest per cluster	Greedy threshold pruning	Ridge — it keeps everything, which is the opposite of the goal
Wide data, p greater than n	PLS or elastic net	PCA with cross-validated component count	VIF — it is undefined when the design matrix is singular

Traps

The dummy variable trap

A full set of one-hot columns sums to a constant, so with an intercept the design matrix is exactly singular. Drop one level, or fit without an intercept. This is perfect collinearity, not near-collinearity, and no pairwise threshold rule will surface it — VIF and the eigenvalue spectrum will.

Pruning on the whole dataset

Selecting features by correlation before splitting leaks the test set into the choice. Feature selection is part of the pipeline and belongs inside the cross-validation loop.

Correlation is not redundancy

Two features at r = 0.9 can still be jointly necessary if a non-linear model uses their difference. Confirm redundancy by removing the feature and measuring the change in validation error, not by reading the matrix.

A very high correlation with the target

An r of 0.98 against the label is usually not a great feature — it is leakage. Check whether the value was actually available at prediction time before celebrating.

Notes

Read the numbers as illustrations of the mechanism, not as measurements of anything real: every sample on this page is drawn from a seeded generator when you open it.

Implementations worth knowing: scipy.stats for Pearson, Spearman and Kendall; dcor for distance correlation; sklearn.feature_selection.mutual_info_regression for the k-nearest-neighbour MI estimator; statsmodels.stats.outliers_influence.variance_inflation_factor for VIF; sklearn.inspection.permutation_importance for the attribution fix.