# Orthogonal Projection

Aug-Nov 2020

## Recap

• Vector space $V$ over a scalar field $F$
• $F$: real field $\mathbb{R}$ or complex field $\mathbb{C}$ in this course
• $m\times n$ matrix A represents a linear map $T:F^n\to F^m$
• dim null $T$ + dim range $T$ = dim $V$
• Solution to $Ax=b$ (if it exists): $u+$ null$(A)$
• Four fundamental subspaces of a matrix
• Column space, row space, null space, left null space
• Eigenvalue $\lambda$ and Eigenvector $v$: $Tv=\lambda v$
• Some linear maps are diagonalizable
• Inner products, norms, orthogonality and orthonormal basis
• Upper triangular matrix for a linear map over an orthonormal basis
• $V=U\oplus U^{\perp}$ for any subspace $U$

## Linear maps onto a subspace

$U$: subspace of $V$

Is there a linear operator $T$ s.t. range$(T)=U$?

Yesâ€¦ there are many!

Basis for $V$: $\{v_1,\ldots,v_n\}$

Define $T$ as mapping each $v_i$ to some $u_i\in U$

Othogonal projection: linear operator taking $v\in V$ into $U$ in a specific way

## Recap: orthogonal complements

$U$: subspace of $V$

$U^{\perp}$: orthogonal complement of $U$

$U^{\perp}=\{v\in V: \langle v,u\rangle=0\text{ for all }u\in U\}$

$V=U\oplus U^{\perp}$

How to find $U^{\perp}$?

Orthonormal basis for $U$: $\{e_1,\ldots,e_m\}$

Extend to orthonormal basis for $V$: $\{e_1,\ldots,e_m,e_{m+1},\ldots,e_n\}$

Orthonormal basis for $U^{\perp}$: $\{e_{m+1},\ldots,e_n\}$

$v=\langle v,e_1\rangle e_1+\cdots+\langle v,e_m\rangle e_m+\langle v,e_{m+1}\rangle e_{m+1}+\cdots+\langle v,e_n\rangle e_n$

$u=\langle v,e_1\rangle e_1+\cdots+\langle v,e_m\rangle e_m\in U$

$u^{\perp}=\langle v,e_{m+1}\rangle e_{m+1}+\cdots+\langle v,e_n\rangle e_n$

$v=u+u^{\perp}$

## Examples

$U=$ span$\{(1,0,0,0),(0,1,0,0)\}$

Orthonormal basis extension

$\{(1,0,0,0),(0,1,0,0),(0,0,1,0),(0,0,0,1)\}$

$U^{\perp}=$ span$\{(0,0,1,0),(0,0,0,1)\}$

$v=(1,0,1,1)=u+u^{\perp}$, where $u=(1,0,0,0)$, $u^{\perp}=(0,0,1,1)$

$U=$ span$\{(1/\sqrt{2},1/\sqrt{2},0,0),(1/\sqrt{2},-1/\sqrt{2},0,0)\}$

Orthonormal basis extension

$\{(1/\sqrt{2},1/\sqrt{2},0,0),(1/\sqrt{2},-1/\sqrt{2},0,0),(0,0,1/\sqrt{2},1/\sqrt{2}),(0,0,1/\sqrt{2},-1/\sqrt{2})\}^1$

$U^{\perp}=\{(0,0,1/\sqrt{2},1/\sqrt{2}),(0,0,1/\sqrt{2},-1/\sqrt{2})\}$

$v=(1,0,1,1)^1=u+u^{\perp}$, where $u=(1,0,0,0)^1$, $u^{\perp}=(0,0,1,1)^1$

## Orthogonal projection

$U$: finite-dimensional subspace of $V$

Orthogonal projection operator $P_U$ maps $v=u+u^{\perp}$, where $u\in U$, $u^{\perp}\in U^{\perp}$, to $u$.

• $P_U$: a linear operator, well-defined by uniqueness of $u$

Example

1. $x\in V$, $x\ne0$, $U=$ span$(x)$

Orthonormal basis for $U$: $\dfrac{x}{\lVert x\rVert}$

$P_U=\frac{\langle v,x\rangle}{\lVert x\rVert^2}x$

1. $x,y\in V$, linearly independent, and $U=$ span$(x,y)$

Orthonormal basis for $U$: $e_1=\dfrac{x}{\lVert x\rVert}$, $e_2=\dfrac{y-\langle y,e_1\rangle e_1}{\lVert y-\langle y,e_1\rangle e_1\rVert}$

$P_U=\langle v,e_1\rangle e_1+\langle v,e_2\rangle e_2$

## Matrices for the projection operator

$V=\mathbb{R}^n$, dot product, standard basis

$U=$ span$\{(x_1,\ldots,x_n)\}$

$P_U=\frac{\langle v,x\rangle}{\lVert x\rVert^2}x = \frac{1}{\lVert x\rVert^2}x(x^Tv)$

$P_U\leftrightarrow \dfrac{1}{\lVert x\rVert^2}xx^T$

$U=$ span$\{e_1=(e_{11},\ldots,e_{1n}),\ldots,e_m=(e_{m1},\ldots,e_{mn})\}$

$e_1,\ldots,e_m$: orthonormal

\begin{align} P_U&=\langle v,e_1\rangle e_1+\cdots+\langle v,e_m\rangle e_m\\ &=(e_1e^T_1+\ldots+e_me^T_m)v \end{align}

$P_U\leftrightarrow e_1e^T_1+\ldots+e_me^T_m$

## Properties

$U$: finite-dimensional subspace of $V$, $v\in V$

1. $P_Uu=u$ for $u\in U$

2. $P_Uw=0$ for $w\in U^{\perp}$

3. range $P_U=U$

4. null $P_U=U^{\perp}$

5. $v-P_Uv\in U^{\perp}$

6. $P^2_U=P_U$

7. $\lVert P_Uv\rVert \le \lVert v\rVert$