Projection: The Shadow of a Vector

Projection asks a beautifully simple question:

How much of $u$ lies in the direction of $v$ ?

Vector u casting an orthogonal shadow onto the line spanned by v. — The projection is the shadow of $u$ on the one-dimensional world spanned by $v$.

For nonzero $v$ ,

\operatorname{proj}_v(u) = \frac{u\cdot v}{v\cdot v}v.

If $\hat v=v/\lVert v\rVert$ is a unit vector, this becomes

\operatorname{proj}_{\hat v}(u)=(u\cdot\hat v)\hat v.

A decomposition

Projection naturally splits a vector into two pieces:

u=u_\parallel+u_\perp,

where $u_\parallel$ lies along $v$ and $u_\perp$ is perpendicular to it.

Vector decomposed into parallel and perpendicular components on a grid. — The original vector is exactly the sum of its parallel component and orthogonal residual.

From shadows to least squares

When a system $Ax=b$ has no exact solution, least squares finds the point $A\hat x$ in the column space of $A$ closest to $b$ . In other words, it projects $b$ onto a subspace.

The residual

r=b-A\hat x

must be perpendicular to every column of $A$ , giving

A^\top(b-A\hat x)=0.

Hence the normal equations:

A^\top A\hat x=A^\top b.

Projection is therefore not a decorative geometric idea. It is hiding inside regression, approximation, signal processing and machine learning.