.. _projectR2: 

Projections Onto a Line 
------------------------------------

.. index:: projection

A nice aspect of linear algebra is its geometric interpretation. That
is, if we consider linear algebra problems in :math:`\mathbb{R}^2` or
:math:`\mathbb{R}^3`, then we can plot the vectors to visually depict
the geometric relationships. This is the case with project, which gives
us a geometric view of over-determined systems. We will begin with the
simplest of examples and later extend the concepts to higher dimensions.

Consider the following over-determined system where the unknown variable
is a scalar.

.. math:: \spalignsys{4x = 2; 2x = 1}

.. math::
   :label: eq-project1

   \bm{a}\,x = \bm{b} \\ 

.. math:: \vector{4; 2} \vector{x} = \vector{2; 1}

Figure :numref:`fig-projectSimple` is a plot of the vectors
:math:`\bm{a}` and :math:`\bm{b}`.

.. _fig-projectSimple:

.. figure:: projectSimple.png
   :align: center
   :width: 40%

   Vectors :math:`\bm{a}` and :math:`\bm{b}` are consistent.

Here :math:`x` is a scalar, not a vector, and we can quickly see that
:math:`x = 1/2` satisfies both rows of equation :eq:`eq-project1`.
But since :math:`\bm{a}` and :math:`\bm{b}` are both vectors, we want to
use a strategy that will extend to matrices. We can multiply both sides
of equation :eq:`eq-project1` by :math:`\bm{a}^T` so that the
vectors turn to scalar dot products.

.. math::
   :label: eq-simpleproject

   \bm{a}^T \bm{a} x = \bm{a}^T \bm{b}

.. math::

   \begin{array}{rl}
             \mat{4 2} \vector{4; 2} x &= \mat{4 2} \vector{2; 1} \\
             x &= \frac{10}{20} = \frac{1}{2}
             \end{array}

Now, let’s extend the problem to a more general case as shown in figure
:numref:`fig-project` where vector :math:`\bm{b}` is not necessarily
in-line with :math:`\bm{a}`. We will find a geometric reason to multiply
both sides of equation :eq:`eq-project1` by :math:`\bm{a}^T`.

.. _fig-project:

.. figure:: project.png
   :align: center
   :width: 40%

   Vector :math:`\bm{p}` is a projection of vector :math:`\bm{b}` onto
   vector :math:`\bm{a}`.

We wish to project a vector :math:`\bm{b}` onto vector :math:`\bm{a}`,
such that the error between :math:`\bm{b}` and the projection is
minimized. The projection, :math:`\bm{p}`, is a scalar multiple of
:math:`\bm{a}`. That is, :math:`\bm{p} = \bm{a}\,x`, where :math:`x` is
the scalar value that we want to find. The error is the vector
:math:`\bm{e} = \bm{b} - \bm{a}\,x`.

The geometry of the problem provides a simple solution for minimizing
the error. The length of the error vector is minimized when it is
perpendicular to :math:`\bm{a}`. Recall that two vectors are
perpendicular (orthogonal) when their dot product is equal to zero.

.. math:: \bm{a}^T\bm{e} = \bm{a}^T(\bm{b} - \bm{a}\,x) = 0

.. math:: \bm{a}^T\bm{a}\,x = \bm{a}^T\bm{b}

.. math:: \boxed{x = \frac{\bm{a}^T\bm{b}}{\bm{a}^T\bm{a}}}

Since :math:`x` is a fraction of two dot products, we can think of the
projection in terms of the angle, :math:`\theta`, between :math:`\bm{a}`
and :math:`\bm{b}`.

.. math::

   \begin{array}{rl}
    x &= \frac{\norm{\bm{a}}\,\norm{\bm{b}} \cos \theta}{\norm{\bm{a}}^2} \\
      &\hfill \\
      &= \frac{\norm{\bm{b}}}{\norm{\bm{a}}} \cos \theta
   \end{array}

The projection is then:

.. math::

   \begin{array}{rl}
     \bm{p} &= \bm{a}\,x \\
      &\hfill \\
   &= \frac{\bm{a}}{\norm{\bm{a}}}\norm{\bm{b}} \cos\theta \\
      &\hfill \\
   &= \bm{a}\,\frac{\bm{a}^T\bm{b}}{\bm{a}^T\bm{a}}
   \end{array}

We can also make a projection matrix, :math:`\bf{P}`, so that any vector
may be projected onto :math:`\bm{a}` by multiplying it by
:math:`\bf{P}`.

.. math:: \boxed{\mathbf{P} = \frac{\bm{a}\,\bm{a}^T}{\bm{a}^T\,\bm{a}}}

.. math:: \bm{p} = \mathbf{P}\,\bm{b}

Note that :math:`\bm{a}\,\bm{a}^T` is here a :math:`2{\times}2` matrix
and :math:`\bm{a}^T\,\bm{a}` is a scalar. Here is an example.

::

   >> b = [3; 3]; a = [4; 2];
   >> P = a*a'/(a'*a)  % Projection Matrix to a
   P =
       0.8000    0.4000
       0.4000    0.2000
   >> p = P*b          % Projection of b onto a
   p =
       3.6000
       1.8000
   >> x = a'*b/(a'*a)  % length of projection
   x =
       0.9000
   >> e = b - p        % error vector
   e =
      -0.6000
       1.2000
   >> p'*e             % near zero dot product to
   ans =               % confirm e is perpendicular to p.
       -4.4409e-16