6.11.1. Projections Onto a Line

A nice aspect of linear algebra is its geometric interpretation. That is, if we consider linear algebra problems in \mathbb{R}^2 or \mathbb{R}^3, then we can plot the vectors to visually depict the geometric relationships. This is the case with project, which gives us a geometric view of over-determined systems. We will begin with the simplest of examples and later extend the concepts to higher dimensions.

Consider the following over-determined system where the unknown variable is a scalar.

\spalignsys{4x = 2; 2x = 1}

(6.29)\bm{a}\,x = \bm{b} \\

\vector{4; 2} \vector{x} = \vector{2; 1}

Figure Fig. 6.24 is a plot of the vectors \bm{a} and \bm{b}.

../../_images/projectSimple.png

Fig. 6.24 Vectors \bm{a} and \bm{b} are consistent.

Here x is a scalar, not a vector, and we can quickly see that x = 1/2 satisfies both rows of equation (6.29). But since \bm{a} and \bm{b} are both vectors, we want to use a strategy that will extend to matrices. We can multiply both sides of equation (6.29) by \bm{a}^T so that the vectors turn to scalar dot products.

(6.30)\bm{a}^T \bm{a} x = \bm{a}^T \bm{b}

\begin{array}{rl} \mat{4 2} \vector{4; 2} x &= \mat{4 2} \vector{2; 1} \\ x &= \frac{10}{20} = \frac{1}{2} \end{array}

Now, let’s extend the problem to a more general case as shown in figure Fig. 6.25 where vector \bm{b} is not necessarily in-line with \bm{a}. We will find a geometric reason to multiply both sides of equation (6.29) by \bm{a}^T.

../../_images/project.png

Fig. 6.25 Vector \bm{p} is a projection of vector \bm{b} onto vector \bm{a}.

We wish to project a vector \bm{b} onto vector \bm{a}, such that the error between \bm{b} and the projection is minimized. The projection, \bm{p}, is a scalar multiple of \bm{a}. That is, \bm{p} = \bm{a}\,x, where x is the scalar value that we want to find. The error is the vector \bm{e} = \bm{b} - \bm{a}\,x.

The geometry of the problem provides a simple solution for minimizing the error. The length of the error vector is minimized when it is perpendicular to \bm{a}. Recall that two vectors are perpendicular (orthogonal) when their dot product is equal to zero.

\bm{a}^T\bm{e} = \bm{a}^T(\bm{b} - \bm{a}\,x) = 0

\bm{a}^T\bm{a}\,x = \bm{a}^T\bm{b}

\boxed{x = \frac{\bm{a}^T\bm{b}}{\bm{a}^T\bm{a}}}

Since x is a fraction of two dot products, we can think of the projection in terms of the angle, \theta, between \bm{a} and \bm{b}.

\begin{array}{rl} x &= \frac{\norm{\bm{a}}\,\norm{\bm{b}} \cos \theta}{\norm{\bm{a}}^2} \\ &\hfill \\ &= \frac{\norm{\bm{b}}}{\norm{\bm{a}}} \cos \theta \end{array}

The projection is then:

\begin{array}{rl} \bm{p} &= \bm{a}\,x \\ &\hfill \\ &= \frac{\bm{a}}{\norm{\bm{a}}}\norm{\bm{b}} \cos\theta \\ &\hfill \\ &= \bm{a}\,\frac{\bm{a}^T\bm{b}}{\bm{a}^T\bm{a}} \end{array}

We can also make a projection matrix, \bf{P}, so that any vector may be projected onto \bm{a} by multiplying it by \bf{P}.

\boxed{\mathbf{P} = \frac{\bm{a}\,\bm{a}^T}{\bm{a}^T\,\bm{a}}}

\bm{p} = \mathbf{P}\,\bm{b}

Note that \bm{a}\,\bm{a}^T is here a 2{\times}2 matrix and \bm{a}^T\,\bm{a} is a scalar. Here is an example.

>> b = [3; 3]; a = [4; 2];
>> P = a*a'/(a'*a)  % Projection Matrix to a
P =
    0.8000    0.4000
    0.4000    0.2000
>> p = P*b          % Projection of b onto a
p =
    3.6000
    1.8000
>> x = a'*b/(a'*a)  % length of projection
x =
    0.9000
>> e = b - p        % error vector
e =
   -0.6000
    1.2000
>> p'*e             % near zero dot product to
ans =               % confirm e is perpendicular to p.
    -4.4409e-16