# The Unapologetic Mathematician

## The Singular Value Decomposition

Now the real and complex spectral theorems give nice decompositions of self-adjoint and normal transformations, respectively. Each one is of a similar form

$\displaystyle\begin{aligned}S&=O\Lambda O^*\\H&=U\Lambda U^*\end{aligned}$

where $O$ is orthogonal, $U$ is unitary, and $\Lambda$ (in either case) is diagonal. What we want is a similar decomposition for any transformation. And, in fact, we’ll get one that even works for transformations between different inner prouct spaces.

So let’s say we’ve got a transformation $M:A\rightarrow B$ (we’re going to want to save $U$ and $V$ to denote transformations). We also have its adjoint $M^*:B\rightarrow A$. Then $M^*M:A\rightarrow A$ is positive-semidefinite (and thus self-adjoint and normal), and so the spectral theorem applies. There must be a unitary transformation $V:A\rightarrow A$ (orthogonal, if we’re working with real vector spaces) so that

$\displaystyle V^*M^*MV=\begin{pmatrix}D&0\\{0}&0\end{pmatrix}$

where $D$ is a diagonal matrix with strictly positive entries.

That is, we can break $A$ up as the direct sum $A=A_1\oplus A_2$. The diagonal transformation $D:A_1\rightarrow A_1$ is positive-definite, while the restriction of $V^*M^*MV$ to $A_2$ is the zero transformation. We will restrict $V$ to each of these subspaces, giving $V_1:A_1\rightarrow A$ and $V_2:A_2\rightarrow A$, along with their adjoints $V_1^*:A\rightarrow A_1$ and $V_2^*:A\rightarrow A_2$. Then we can write

$\displaystyle\begin{pmatrix}V_1^*\\V_2^*\end{pmatrix}M^*M\begin{pmatrix}V_1&V_2\end{pmatrix}=\begin{pmatrix}V_1^*M^*MV_1&V_1^*M^*MV_2\\V_2^*M^*MV_1&V_2^*M^*MV_1\end{pmatrix}=\begin{pmatrix}D&0\\{0}&0\end{pmatrix}$

From this we conclude both that $V_1^*M^*MV_1=D$ and that $MV_2=0$. We define $U_1=MV_1D^{-\frac{1}{2}}:A_1\rightarrow B$, where we get the last matrix by just taking the inverse of the square root of each of the diagonal entries of $D$ (this is part of why diagonal transformations are so nice to work with). Then we can calculate

$\displaystyle\begin{aligned}U_1D^\frac{1}{2}V_1^*&=MV_1D^{-\frac{1}{2}}D^\frac{1}{2}V_1^*\\&=MV_1V_1^*\\&=MV_1V_1^*+MV_2V_2^*\\&=M\left(V_1V_1^*+V_2V_2^*\right)\\&=MVV^*=M\end{aligned}$

This is good, but we don’t yet have unitary matrices in our decomposition. We do know that $V_1^*V_1=1_{A_1}$, and we can check that

$\displaystyle\begin{aligned}U_1^*U_1&=\left(MV_1D^{-\frac{1}{2}}\right)^*MV_1D^{-\frac{1}{2}}\\&=D^{-\frac{1}{2}}V_1^*M^*MV_1D^{-\frac{1}{2}}\\&=D^{-\frac{1}{2}}DD^{-\frac{1}{2}}=1_{A_1}\end{aligned}$

Now we know that we can use $V_2:A_2\rightarrow A$ to “fill out” $V_1$ to give the unitary transformation $V$. That is, $V_1^*V_1=1_{A_1}$ (as we just noted), $V_2^*V_2=1_{A_2}$ (similarly), $V_1^*V_2$ and $V_2^*V_1$ are both the appropriate zero transformation, and $V_1V_1^*+V_2V_2^*=1_A$. Notice that these are exactly stating that the adjoints $V_1^*$ and $V_2^*$ are the projection operators corresponding to the inclusions $V_1$ and $V_2$ in a direct sum representation of $A$ as $A_1\oplus A_2$. It’s clear from general principles that there must be some projections, but it’s the unitarity of $V$ that makes the projections be exactly the adjoints of the inclusions.

What we need to do now is to similarly fill out $U_1$ by supplying a corresponding $U_2:B_2\rightarrow B$ that will similarly “fill out” a unitary transformation $U$. But we know that we can do this! Pick an orthonormal basis of $A_1$ and hit it with $U_1$ to get a bunch of orthonormal vectors in $B$ (orthonormal because $U_1^*U_1=1_{A_1}$. Then fill these out to an orthonormal basis of all of $B$. Just set $B_2$ to be the span of all the new basis vectors, which is the orthogonal complement of the image of $U_1$, and let $U_2$ be the inclusion of $B_2$ into $B$. We can then combine to get a unitary transformation

$\displaystyle U=\begin{pmatrix}U_1&U_2\end{pmatrix}$

Finally, we define

$\displaystyle\Sigma=\begin{pmatrix}D^\frac{1}{2}&0\\{0}&0\end{pmatrix}$

where there are as many zero rows in $\Sigma$ as we needed to add to fill out the basis of $B$ (the dimension of $B_2$). I say that $U\Sigma V^*$ is our desired decomposition. Indeed, we can calculate

$\displaystyle\begin{aligned}U\Sigma V^*&=\begin{pmatrix}U_1&U_2\end{pmatrix}\begin{pmatrix}D^\frac{1}{2}&0\\{0}&0\end{pmatrix}\begin{pmatrix}V_1^*\\V_2^*\end{pmatrix}\\&=\begin{pmatrix}U_1D^\frac{1}{2}&0\end{pmatrix}\begin{pmatrix}V_1^*\\V_2^*\end{pmatrix}\\&=U_1D^\frac{1}{2}V_1^*=M\end{aligned}$

where $U$ and $V$ are unitary on $B$ and $A$, respectively, and $\Sigma$ is a “diagonal” transformation (not strictly speaking in the case where $A$ and $B$ have different dimensions).

### Like this:

Posted by John Armstrong | Algebra, Linear Algebra

## 3 Comments	»

1. [...] Meaning of the SVD We spent a lot of time yesterday working out how to write down the singular value decomposition of a transformation , [...]

Pingback by | August 18, 2009 | Reply

2. [...] Okay, let’s take the singular value decomposition and do something really neat with it. Specifically, we’ll start with an endomorphism and [...]

Pingback by | August 19, 2009 | Reply

3. [...] any linear transformation between two inner product spaces we can find orthonormal bases giving the singular value decomposition. There are two unitary (or orthogonal) transformations and and a “diagonal” [...]

Pingback by | August 24, 2009 | Reply

« Previous | Next »

## About this weblog

This is mainly an expository blath, with occasional high-level excursions, humorous observations, rants, and musings. The main-line exposition should be accessible to the “Generally Interested Lay Audience”, as long as you trace the links back towards the basics. Check the sidebar for specific topics (under “Categories”).

I’m in the process of tweaking some aspects of the site to make it easier to refer back to older topics, so try to make the best of it for now.
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 72, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "github"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "english_confidence": 0.9138296246528625, "perplexity_flag": "head"}