.. _svdGeometry:

The Geometry of the SVD
=======================

As we have seen before, we can visualize the geometry of linear algebra
equations in :math:`\mathbb{R}^2` or :math:`\mathbb{R}^3` and the
relationships hold in higher dimensions where we can not see the
relationships in a diagram. Recall from our introduction to eigenvalue
problems at the beginning of :ref:`eig` that multiplication of a
matrix and a vector usually stretches and rotates the vector. As before,
we want to factor out the stretching and rotating components of the
matrix. For a :math:`m{\times}n` matrix :math:`\bf{A}` and a
:math:`n{\times}1` unit vector :math:`\bm{v}`, we have

.. math::
   :label: eq-svd-vect

   \mathbf{A}\,\bm{v} = \sigma\,\bm{u},

where :math:`\sigma` is a scalar that is the length of the resulting vector and
:math:`\bm{u}` is a unit vector that shows the direction of the resulting
vector. There are :math:`n` vector–matrix relationships like equation
:eq:`eq-svd-vect` that make the SVD factorization of a :math:`m{\times}n`
matrix.

Consider the following two orthogonal unit vectors that are multiplied
by a yet unknown matrix.

.. math::

   \bm{v}_1 = \frac{\sqrt{2}}{2}\vector{-1; 1} \qquad
   \bm{v}_2 = \frac{\sqrt{2}}{2}\vector{1; 1}

The product of each vector with the :math:`\bf{A}` matrix is represented
by two perpendicular vectors :math:`\sigma_i\,\bm{u}_i`, where the
:math:`\sigma_i` are scalar values called the singular values, and the
:math:`\bm{u}_i` are unit vectors that are rotated from the original
:math:`\bm{v}_i` vectors but are still perpendicular to each other. In
this example, the resulting vectors from multiplication with
:math:`\bf{A}` are:

.. math::

   \sigma_1\,\bm{u}_1 = \vector{-1; \sqrt{3}} \qquad
       \sigma_2\,\bm{u}_2 = \frac{1}{2}\vector{\sqrt{3}; 1}.

After finding the unit vectors, we have:

.. math::

   \sigma_1 = 2, \; \bm{u}_1 = \frac{1}{2}\vector{-1; \sqrt{3}} \quad
       \sigma_2 = 1, \;\bm{u}_2 = \frac{1}{2}\vector{\sqrt{3}; 1}.

We can apply equation :eq:`eq-svd-vect` to find the
two columns of :math:`\bf{A}`.

.. math::

   \begin{aligned}
           \mathbf{A} &= \frac{1}{2}\mat{-1 \sqrt{3}; \sqrt{3} 1}\, 
           \mat{2 0; 0 1} \, \frac{\sqrt{2}}{2}\mat{-1 1; 1 1} \\
            &= \frac{\sqrt{2}}{4}\mat{{\left(2 + \sqrt{3}\right)} 
                       {\left(-2 + \sqrt{3}\right)}; 
                    {\left(1 - 2\,\sqrt{3}\right)} 
                       {\left(1 + 2\,\sqrt{3}\right)}} \approx
                   \mat{1.3195 -0.0947; -0.8712 1.5783}
       \end{aligned}

This example shows the relationship between the SVD factors and the
impact of multiplying a matrix with a vector. Of course, most of the
time we approach the SVD from the opposite perspective where we have a
matrix and want to find the three SVD factors :math:`\bf{U}`,
:math:`\bf{\Sigma}`, and :math:`\mathbf{V}^H`.