Orthogonal projection · aiwiki.page

Orthogonal projection is an operation in linear algebra that separates a vector into a component belonging to a specified linear subspace and a component perpendicular to it. The first component is the projection; it is also the closest vector in that subspace, with distance measured by the norm induced by an inner product. The construction applies to finite-dimensional inner-product spaces and, more generally, to closed subspaces of a Hilbert space. (terrytao.wordpress.com)

Definition and geometric meaning

Let $V$ be a finite-dimensional inner-product vector space, and let $S\subseteq V$ be a subspace. The orthogonal projection of $x\in V$ onto $S$ , written $P_Sx$ , is the unique vector satisfying

P_Sx\in S,\qquad \langle x-P_Sx,s\rangle=0 \quad\text{for every }s\in S.

Thus,

x=P_Sx+(x-P_Sx).

The residual belongs to the orthogonal complement $S^\perp$ , the set of vectors perpendicular to every vector in $S$ . Projection preserves vectors already in $S$ and sends vectors in $S^\perp$ to zero. (ocw.mit.edu)

In Euclidean geometry, projection onto a line or plane is the foot of the perpendicular drawn from the original point. Its nearest-point property follows from the Pythagorean theorem: for any $s\in S$ ,

\|x-s\|^2 =\|x-P_Sx\|^2+\|P_Sx-s\|^2.

The second term is nonnegative and vanishes only when $s=P_Sx$ , establishing both optimality and uniqueness. (terrytao.wordpress.com)

Explicit formulas

For a nonzero real vector $u$ , projection onto its linear span is

P_ux=\frac{u^{\mathsf T}x}{u^{\mathsf T}u}\,u.

For example, projecting $x=(3,1)$ onto the line spanned by $u=(1,1)$ gives $P_ux=(2,2)$ . The residual $(1,-1)$ has zero inner product with $u$ . Rescaling $u$ by any nonzero real number leaves the projection unchanged because the target line is unchanged. (ocw.mit.edu)

If $q_1,\ldots,q_k$ form an orthonormal basis of $S$ , then

P_Sx=\sum_{j=1}^{k}\langle x,q_j\rangle q_j,

using the convention that the inner product is linear in its first argument. Writing these basis vectors as columns of a matrix $Q$ gives

P_S=QQ^*, \qquad Q^*Q=I_k.

Here $Q^*$ denotes conjugate transpose; for real matrices it is the ordinary transpose. Projection therefore consists of finding coordinates along orthonormal directions and reconstructing the corresponding vector in $S$ . (ocw.mit.edu)

Operator and matrix properties

An orthogonal projector, regarded as a linear map from the ambient space to itself, satisfies

P^2=P,\qquad P^*=P.

The first identity expresses idempotence: projecting twice has the same effect as projecting once. The second expresses self-adjointness. Together, these properties characterize orthogonal projectors among bounded operators on a Hilbert space. Their image is the target subspace, and their kernel is its orthogonal complement. (mathweb.ucsd.edu)

Consequently, in finite dimensions the only possible eigenvalues are $0$ and $1$ : vectors in the target subspace have eigenvalue $1$ , while vectors in its orthogonal complement have eigenvalue $0$ . The rank equals the target subspace’s dimension. Also, $I-P$ , with $I$ the identity matrix, projects onto the orthogonal complement. These statements follow directly from the decomposition into the two subspaces. (mathweb.ucsd.edu)

Projection does not increase length:

\|Px\|\leq\|x\|.

It should not be confused with an orthogonal matrix, which preserves lengths and is invertible. Projection onto a proper subspace discards information. An idempotent operator lacking self-adjointness may instead be an oblique projection, whose discarded component need not be perpendicular to its image. (terrytao.wordpress.com)

Least squares and computation

Suppose the columns of a real matrix $A$ are linearly independent and span $S$ . Then

P_S=A(A^{\mathsf T}A)^{-1}A^{\mathsf T}.

The matrix $A^{\mathsf T}A$ is a Gram matrix. To project a vector $b$ , solve the normal equations

A^{\mathsf T}A\widehat c=A^{\mathsf T}b,

and set $p=A\widehat c$ . These equations express the condition that $b-p$ is perpendicular to every column of $A$ . (ocw.mit.edu)

This is the geometry underlying ordinary least squares and linear regression: the fitted response vector is the projection of the observed response onto the design matrix’s column space. If the columns are dependent, the fitted vector remains unique even when the coefficient vector is not. For any finite matrix,

P_S=AA^+,

where $A^+$ is the Moore–Penrose pseudoinverse. (ocw.mit.edu)

In numerical linear algebra, a QR decomposition supplies orthonormal columns $Q$ , allowing projection as $Q(Q^*b)$ without explicitly constructing the full projector. QR methods also avoid the increased numerical sensitivity associated with forming $A^*A$ , whose condition number is squared relative to $A$ when $A$ has full column rank. (ocw.mit.edu)

Infinite-dimensional extension and related projections

The Hilbert-space projection theorem states that every vector has a unique orthogonal decomposition relative to a closed subspace $S$ . Closedness matters: if $S$ is not closed, a vector in its closure but outside $S$ has distance zero from $S$ , yet no nearest point in $S$ . Projection onto the closure still exists. (terrytao.wordpress.com)

A broader nearest-point theorem applies to every nonempty closed convex set in a Hilbert space. Such a projection is unique and nonexpansive, but generally is not linear. Projection onto an affine subspace $a+S$ , for example, is

P_{a+S}(x)=a+P_S(x-a),

which is an affine rather than necessarily linear map. This distinguishes nearest-point projection onto general sets from orthogonal projection onto linear subspaces. (math.ucla.edu)