.. _projectR2:

Projections Onto a Line
-----------------------

A nice aspect of linear algebra is its geometric interpretation. That
is, if we consider linear algebra problems in :math:`\mathbb{R}^2` or
:math:`\mathbb{R}^3`, then we can plot the vectors to depict the
geometric relationships visually. Here, we consider vector projection,
which gives us a geometric view of over-determined systems. We will
begin with a simple example and later extend the concepts to higher
dimensions.

Consider the following over-determined system where the unknown variable
is a scalar.

.. math:: \left\{\begin{matrix}
             4x  & = & 2 \\ 
             2x  & = & 1
          \end{matrix}\right.

.. math::
   :label: eq-project1

   \bm{a}\,x = \bm{b} \\ 

.. math:: \begin{bmatrix}
            4 \\ 2 
          \end{bmatrix} \begin{bmatrix}
                          x 
                        \end{bmatrix} = \begin{bmatrix}
                                       2 \\ 1 
                                     \end{bmatrix}

Figure :numref:`fig-projectSimple` is a plot of the vectors
:math:`\bm{a}` and :math:`\bm{b}`.

.. _fig-projectSimple:

.. figure:: projectSimple.png
   :figclass: center-caption
   :align: center
   :width: 40%
   :alt: Vectors a and b are consistent.

   Vectors :math:`\bm{a}` and :math:`\bm{b}` are consistent.

Here, :math:`x` is a scalar, not a vector, and we can quickly see that
:math:`x = 1/2`. But since :math:`\bm{a}` and :math:`\bm{b}` are
vectors, we want to use a strategy that we can extend to matrices. We
can multiply both sides of equation :eq:`eq-project1` by
:math:`\bm{a}^T` to turn the vectors into scalar dot products.

.. math::
   :label: eq-simpleproject

   \bm{a}^T \bm{a} x = \bm{a}^T \bm{b}

.. math::

   \begin{array}{rl}
             \begin{bmatrix}
                4  & 2
             \end{bmatrix} \begin{bmatrix}
                         4 \\ 2 
                       \end{bmatrix} x &= \begin{bmatrix}
                                             4  & 2
                                          \end{bmatrix} \begin{bmatrix}
                                                      2 \\ 1 
                                                    \end{bmatrix} \\
             x &= \frac{10}{20} = \frac{1}{2}
             \end{array}

Now, let’s extend the problem to a more general case as shown in figure
:numref:`fig-project` where vector :math:`\bm{b}` is not necessarily
inline with :math:`\bm{a}`. We will find a geometric reason to multiply
both sides of equation :eq:`eq-project1` by :math:`\bm{a}^T`.

.. _fig-project:

.. figure:: project.png
   :figclass: center-caption
   :align: center
   :width: 40%
   :alt: Vector p is a projection of vector b onto vector a.

   Vector :math:`\bm{p}` is a projection of vector :math:`\bm{b}` onto vector
   :math:`\bm{a}`. 

We wish to project the vector :math:`\bm{b}` onto
vector :math:`\bm{a}`, such that the error between :math:`\bm{b}` and
the projection is minimized. The projection, :math:`\bm{p}`, is a scalar
multiple of :math:`\bm{a}`. That is, :math:`\bm{p} = \bm{a}\,x`, where
:math:`x` is the scalar value that we want to find. The error is the
vector :math:`\bm{e} = \bm{b} - \bm{a}\,x`.

The geometry of the problem provides a simple solution for minimizing
the error. The length of the error vector is minimized when it is
perpendicular to :math:`\bm{a}`. Recall that two vectors are
perpendicular (orthogonal) when their dot product equals zero.

.. math:: \bm{a}^T\bm{e} = \bm{a}^T(\bm{b} - \bm{a}\,x) = 0

.. math:: \bm{a}^T\bm{a}\,x = \bm{a}^T\bm{b}

.. math::
   :label: eq-vectorProject

     \boxed{x = \frac{\bm{a}^T\bm{b}}{\bm{a}^T\bm{a}}}

Since :math:`x` is a fraction of two dot products, we can think of the
projection in terms of the angle, :math:`\theta`, between :math:`\bm{a}`
and :math:`\bm{b}`.

.. math::

   \begin{array}{rl}
    x &= \frac{\norm{\bm{a}}\,\norm{\bm{b}} \cos \theta}{\norm{\bm{a}}^2} \\
      &\hfill \\
      &= \frac{\norm{\bm{b}}}{\norm{\bm{a}}} \cos \theta
   \end{array}

The projection is then:

.. math::

   \begin{array}{rl}
     \bm{p} &= \bm{a}\,x \\
      &\hfill \\
   &= \frac{\bm{a}}{\norm{\bm{a}}}\norm{\bm{b}} \cos\theta \\
      &\hfill \\
   &= \bm{a}\,\frac{\bm{a}^T\bm{b}}{\bm{a}^T\bm{a}}
   \end{array}

We can also make a projection matrix, :math:`\bf{P}`, so that any vector
may be projected onto :math:`\bm{a}` by multiplying it by
:math:`\bf{P}`.

.. math:: \boxed{\mathbf{P} = \frac{\bm{a}\,\bm{a}^T}{\bm{a}^T\,\bm{a}}}

.. math:: \bm{p} = \mathbf{P}\,\bm{b}

Note that :math:`\bm{a}\,\bm{a}^T` is here a :math:`2{\times}2` matrix
and :math:`\bm{a}^T\,\bm{a}` is a scalar. Here is an example.

::

    In [16]: b = np.array([[3, 3]]).T; a = np.array([[4, 2]]).T

    In [17]: P = (a @ a.T)/(a.T @ a) # Projection Matrix to a

    In [18]: print(P)
    [[0.8 0.4]
     [0.4 0.2]]

    In [19]: p = P @ b; print(p)     # Projection of b onto a
    [[3.6]
     [1.8]]

    In [20]: x = (a.T @ b)/(a.T @ a); print(x) # length of projection
    [[0.9]]

    In [21]: e = b - p; print(e)     # error vector
    [[-0.6]
     [ 1.2]]

    In [22]: print(p.T @ e)          # Near zero dot product
    [[-2.66453526e-15]]              # confirms e is perpendicular to p.

.. index:: projection