# Singular Values and Vectors of a Linear Map

Aug-Nov 2020

## Recap

• Vector space $V$ over a scalar field $F= \mathbb{R}$ or $\mathbb{C}$
• $m\times n$ matrix A represents a linear map $T:F^n\to F^m$
• dim null $T+$ dim range $T=$ dim $V$
• Solution to $Ax=b$ (if it exists): $u+$ null$(A)$
• Four fundamental subspaces of a matrix
• Column space, row space, null space, left null space
• Eigenvalue $\lambda$ and Eigenvector $v$: $Tv=\lambda v$
• There is a basis w.r.t. which a linear map is upper-triangular
• If there is a basis of eigenvectors, linear map is diagonal w.r.t. it
• Inner products, norms, orthogonality and orthonormal basis
• There is an orthonormal basis w.r.t. which a linear map is upper-triangular
• Orthogonal projection: distance from a subspace
• Adjoint of a linear map: $\langle Tv,w\rangle=\langle v,T^*w\rangle$
• null $T=$ $($range $T^*)^{\perp}$
• Self-adjoint: $T=T^*$, Normal: $TT^*=T^*T$
• Complex/real spectral theorem: $T$ is normal/self-adjoint $\leftrightarrow$ orthonormal basis of eigenvectors
• Positive operators: self-adjoint with non-negative eigenvalues
• Isometries: normal with absolute value 1 eigenvalues

## Singular values of a linear map

$T:V\to W$, linear map

Singular values of $T$ are the eigenvalues of $\sqrt{T^*T}$.

1. $T^*T:V\to V$: (self-adjoint) positive operator.

2. Spectral theorem: $T^*T\leftrightarrow \lambda_1e_1\overline{e^T_1}+\cdots+\lambda_ne_n\overline{e^T_n}$ Eigenvalues: $\lambda_1\ge\cdots\ge\lambda_n$
$\{e_1,\ldots,e_n\}$: orthonormal basis, $n=$ dim $V$

3. $T^*T$ has a unique positive square root $\sqrt{T^*T}$ $\sqrt{T^*T}\leftrightarrow \sqrt{\lambda_1}e_1\overline{e^T_1}+\cdots+\sqrt{\lambda_n}e_n\overline{e^T_n}$

4. Singular values of $T$: $\sqrt{\lambda_1}\ge\cdots\ge\sqrt{\lambda_n}$

## Singular vectors of a linear map

$T:V\to W$, linear map

$T^*T:V\to V$, self-adjoint

Right-singular vectors of $T$ are an orthonormal eigenvector basis vectors of $T^*T$

Note: Right-singular vectors are vectors in $V$

$TT^*:W\to W$, self-adjoint

Left-singular vectors of $T$ are an orthonormal eigenvector basis vectors of $TT^*$

Note: Left-singular vectors are vectors in $W$

## Example: $2\times 2$

$A=\begin{bmatrix}1&2\\3&4\end{bmatrix}$ (standard basis)

Eigenvalues: $5.372$, $-0.372$; Eigenvectors: $(0.415,0.909)$, $(0.824,-0.566)$

$\begin{bmatrix}1&2\\3&4\end{bmatrix}\begin{bmatrix}0.415\\0.909\end{bmatrix}=5.372\begin{bmatrix}0.415\\0.909\end{bmatrix},\quad\begin{bmatrix}1&2\\3&4\end{bmatrix}\begin{bmatrix}0.824\\-0.566\end{bmatrix}=-0.372\begin{bmatrix}0.824\\-0.566\end{bmatrix}$

$A^TA=\begin{bmatrix}10&14\\14&20\end{bmatrix}$, $AA^T=\begin{bmatrix}5&11\\11&25\end{bmatrix}$

Eigenvalues of $A^TA$ and $AA^T$: $29.866$, $0.134$

Singular values of $A$: $\sigma_1=5.465$, $\sigma_2=0.366$

Right-singular vectors of $A$: $e_1=(0.576,0.817)$, $e_2=(0.817,-0.576)$

Left-singular vectors of $A$: $f_1=(0.404,0.914)$, $f_2=(-0.914,0.404)$

$\begin{bmatrix}1&2\\3&4\end{bmatrix}\begin{bmatrix}0.576\\0.817\end{bmatrix}=5.465\begin{bmatrix}0.404\\0.914\end{bmatrix},\quad\begin{bmatrix}1&2\\3&4\end{bmatrix}\begin{bmatrix}0.817\\-0.576\end{bmatrix}=0.366\begin{bmatrix}-0.914\\0.404\end{bmatrix}$

Observation: $Ae_1=\sigma_1f_1$ and $Ae_2=\sigma_2f_2$ (This is not an accident)

## Example: $2\times 3$

$A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix}$ (eigenvalues undefined)

$A^TA=\begin{bmatrix}17&22&27\\22&29&36\\27&36&45\end{bmatrix}$, $AA^T=\begin{bmatrix}14&32\\32&77\end{bmatrix}$

Eigenvalues of $A^TA$: $90.403$, $0.597$, $0$

Eigenvalues of $AA^T$: $90.403$, $0.597$

Singular values of $A$: $\sigma_1=9.508$, $\sigma_2=0.773$, $\sigma_3=0$

Right-singular vectors of $A$: $e_1=(0.429,0.566,0.704)$, $e_2=(0.805,0.112,-0.581)$, $e_3=(0.408,-0.816,0.408)$

Left-singular vectors of $A$: $f_1=(0.386,0.922)$, $f_2=(-0.922,0.386)$

$\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix}\begin{bmatrix}0.429\\0.566\\0.704\end{bmatrix}=9.508\begin{bmatrix}0.386\\0.922\end{bmatrix},\quad\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix}\begin{bmatrix}0.805\\0.112\\-0.581\end{bmatrix}=0.773\begin{bmatrix}-0.922\\0.386\end{bmatrix}$

$Ae_1=\sigma_1f_1$, $Ae_2=\sigma_2f_2$ and $Ae_3=0$

## Null/range of $T$, $T^*$, $T^*T$, $\sqrt{T^*T}$ and $TT^*$

$T:V\to W$, $T^*T:V\to V$, $\sqrt{T^*T}:V\to V$, $TT^*:W\to W$

1. null $T=$ $($range $T^*)^{\perp}$
null $T^*=$ $($range $T)^{\perp}$
dim range $T=$ dim range $T^*$

2. null $T^*T=$ null $T$
range $T^*T=$ $($null $T^*T)^{\perp}=$ $($null $T)^{\perp}=$ range $T^*$
dim range $T^*T=$ dim range $T^*=$ dim range $T$

3. null $\sqrt{T^*T}=$ null $T^*T=$ null $T$
range $\sqrt{T^*T}=$ $($null $\sqrt{T^*T})^{\perp}=$ $($null $T)^{\perp}$
dim range $\sqrt{T^*T}=$ dim range $T$

4. null $TT^*=$ null $T^*=$ $($range $TT^*)^{\perp}$
range $TT^*=$ $($null $T^*)^{\perp}=$ range $T$