☕biscuits in tea
☕ biscuits in teaMathematicsCovariance: How Variables Move Together↑ Top
Essay

Covariance: How Variables Move Together

From scatter clouds to covariance ellipses, principal directions and positive semidefinite matrices.

Variance asks how one variable spreads. Covariance asks whether two variables move together.

For random variables X,YX,Y,

Cov⁡(X,Y)=E[(X−μX)(Y−μY)].\operatorname{Cov}(X,Y) = \mathbb E[(X-\mu_X)(Y-\mu_Y)].
Three scatter clouds showing positive, near-zero, and negative covariance.
The orientation of a cloud tells the sign of covariance; its spread tells us much more.

For a random vector X∈RnX\in\mathbb R^n,

Σ=E[(X−μ)(X−μ)⊤].\Sigma = \mathbb E[(X-\mu)(X-\mu)^\top].

Why covariance matrices are PSD

Take any vector aa. Then

a⊤Σa=Var⁡(a⊤X)≥0.a^\top\Sigma a = \operatorname{Var}(a^\top X) \ge 0.

So Σ\Sigma is positive semidefinite.

This is not merely algebra: a⊤Σaa^\top\Sigma a is the variance of the data when viewed along direction aa.

Scatter cloud with covariance ellipse and principal eigenvector axes.
The eigenvectors of the covariance matrix give the ellipse's principal axes; eigenvalues encode spread along them.

And suddenly covariance connects directly to eigenvectors and PCA.