9.1. The Geometry of the SVD

As we have seen before, we can visualize the geometry of linear algebra equations in \mathbb{R}^2 or \mathbb{R}^3 and the relationships hold in higher dimensions where we can not see the relationships in a diagram. Recall from our introduction to eigenvalue problems at the beginning of Application of Eigenvalues and Eigenvectors that multiplication of a matrix and a vector usually stretches and rotates the vector. As before, we want to factor out the stretching and rotating components of the matrix. For a m{\times}n matrix \bf{A} and a n{\times}1 unit vector \bm{v}, we have

(9.1)\mathbf{A}\,\bm{v} = \sigma\,\bm{u},

where \sigma is a scalar that is the length of the resulting vector and \bm{u} is a unit vector that shows the direction of the resulting vector. There are n vector–matrix relationships like equation (9.1) that make the SVD factorization of a m{\times}n matrix.

Consider the following two orthogonal unit vectors that are multiplied by a yet unknown matrix.

\bm{v}_1 = \frac{\sqrt{2}}{2}\vector{-1; 1} \qquad \bm{v}_2 = \frac{\sqrt{2}}{2}\vector{1; 1}

The product of each vector with the \bf{A} matrix is represented by two perpendicular vectors \sigma_i\,\bm{u}_i, where the \sigma_i are scalar values called the singular values, and the \bm{u}_i are unit vectors that are rotated from the original \bm{v}_i vectors but are still perpendicular to each other. In this example, the resulting vectors from multiplication with \bf{A} are:

\sigma_1\,\bm{u}_1 = \vector{-1; \sqrt{3}} \qquad \sigma_2\,\bm{u}_2 = \frac{1}{2}\vector{\sqrt{3}; 1}.

After finding the unit vectors, we have:

\sigma_1 = 2, \; \bm{u}_1 = \frac{1}{2}\vector{-1; \sqrt{3}} \quad \sigma_2 = 1, \;\bm{u}_2 = \frac{1}{2}\vector{\sqrt{3}; 1}.

We can apply equation (9.1) to find the two columns of \bf{A}.

\begin{aligned} \mathbf{A} &= \frac{1}{2}\mat{-1 \sqrt{3}; \sqrt{3} 1}\, \mat{2 0; 0 1} \, \frac{\sqrt{2}}{2}\mat{-1 1; 1 1} \\ &= \frac{\sqrt{2}}{4}\mat{{\left(2 + \sqrt{3}\right)} {\left(-2 + \sqrt{3}\right)}; {\left(1 - 2\,\sqrt{3}\right)} {\left(1 + 2\,\sqrt{3}\right)}} \approx \mat{1.3195 -0.0947; -0.8712 1.5783} \end{aligned}

This example shows the relationship between the SVD factors and the impact of multiplying a matrix with a vector. Of course, most of the time we approach the SVD from the opposite perspective where we have a matrix and want to find the three SVD factors \bf{U}, \bf{\Sigma}, and \mathbf{V}^H.