9.5. Fundamental Matrix Subspaces¶

There are four possible fundamental vector subspaces of an $m{\times}n$ matrix $\bf{A}$ . They are called the column space, the null space, the row space, and the left null space. The subspaces are found from rank determined regions of the SVD factors. The regions are identified in figure Fig. 9.8 as $\mathbf{\tilde{U}}$ , $\mathbf{U}_{null}$ , $\mathbf{\tilde{V}}^T$ , and $\mathbf{V}_{null}^T$ .

Fig. 9.8 The columns of $\bf{U}$ and rows of $\mathbf{V}^T$ from the SVD constitute the four fundamental matrix subspaces of the matrix. Which subspace a row or column is part of depends on if it is multiplied by a nonzero or zero singular value.

Column Space: $\mathbf{\tilde{U}}$ : The first $r$ columns of $\bf{U}$ are the basis vectors for the vector space spanned by the columns of $\bf{A}$ .

Null Space: $\mathbf{V}_{null}^T$ : Rows $r$ to $n$ of $\mathbf{V}^T$ are the transpose of the basis vectors for the space spanned by the column vectors $\bm{v}$ such that $\mathbf{A}\,\bm{v} = \bm{0}$ . Note that square, full-rank matrices do not have a null space. MATLAB’s null function returns these vectors.

Row Space: $\mathbf{\tilde{V}}^T$ : The first $r$ rows of $\mathbf{V}^T$ are the basis vectors for the space spanned by the rows of $\bf{A}$ .

Left Null Space: $\mathbf{U}_{null}$ : Columns $r$ to $m$ of $\bf{U}$ are the transpose of the basis vectors for the space spanned by the row vectors $\bm{u}$ such that $\bm{u}\,\mathbf{A} = \bm{0}$ . Note that square, full-rank matrices do not have a left null space.

9.5.1. Column Space¶

The column space of a matrix $\bf{A}$ , which we denote as $\text{Col}( \bf{A} )$ , is the vector space spanned by all independent columns of $\bf{A}$ . If $\bf{A}$ is a square full rank matrix, then it is all of the columns of $\bf{A}$ . For singular or under-determined matrices the column space is the set of columns with nonzero pivots during elimination. The vectors in the column space can also be referred to as basis vectors. Of course, an orthogonal set of basis vectors is preferred and can be found using the algorithms mentioned in Finding Orthogonal Basis Vectors.

Rank and Column Space Dimension

The rank of the matrix is the same as the dimension of the column space, which is the number of columns in the column space.

When there is an exact solution to the equation $\mathbf{A}\,\bm{x} = \bm{b}$ , we say that $\bm{b}$ is “in the column space of $\bf{A}$ ” because it is a linear combination of the independent columns of $\bf{A}$ . The vector $\bm{x}$ tells us the coefficients for the linear combination of the columns that are needed to get to $\bm{b}$ .

Here is a singular matrix example, which shows a case where the column space is a subset of the columns of $\bf{A}$ .

>> A(:,3) = -2*A(:,1) - A(:,2)
A =
    -1     4    -2
     3     0    -6
     6     2   -14
>> rank(A)
ans =
    2
>> rref(A)
ans =
    1     0    -2
    0     1    -1
    0     0     0

>> [U, ~, ~] = svd(A);
>> rank([U(:,1:2) A(:,1:2)])
ans =
     2

Notice from the output of rref that the first two columns are pivot columns. Thus we can see, as was designed from constructing the singular matrix, that the first two columns span the column space.

$\text{Col}( \bf{A} ) = \text{Span} \left\{ \vector{-1; 3; 6} \:,\: \vector{4; 0; 2} \right\}$

Then from the SVD, the first two columns of $\bf{U}$ provides an orthogonal basis for the column space. As we did in Exact Solution or Approximation?, we verified that the first two columns of $\bf{U}$ and $\bf{A}$ are in the same column space because the rank of the augmented matrix of the four columns is still two. The subspace here is a two dimensional plane that is slanted relative to the $\mathbb{R}^3$ coordinate system.

9.5.2. Null Space¶

The null space (right null space) of a matrix $\bf{A}$ , which we denote as $\text{Null}( \bf{A} )$ , is the vector space spanned by all column vectors $\bm{x}$ that satisfy the matrix equation $\mathbf{A}\,\bm{x} = \bm{0}$ . Square, singular matrices and under-determined matrices have a null space. The number of vectors in the null space is the number of dependent columns (size(A, 2) - rank(A)). If $\bf{A}$ is a square full rank matrix, then the null space is an empty set.

The null function returns the normalized basis vectors of a matrix’s null space. MATLAB uses the SVD to find the null space from the last $(n - r)$ columns of $\bf{V}$ corresponding to the singular values equal to zero.

The null space vectors can also be found by elimination from the reduced row echelon form of the matrix as was done in Reduced Row Echelon Form.

Here is an example of a singular matrix.

% Find a random, singular matrix
>> A(:,3) = 4*A(:,1) + 2*A(:,2)
A =
    1     7    18
    1     5    14
    6    -2    20

>> rref(A)
ans =
    1     0     4
    0     1     2
    0     0     0

The third column from the RREF shows the dependent relationship of the columns of $\bf{A}$ . The third column is four of the first column plus two of the second column.

$4\,\bm{a}_1 + 2\,\bm{a}_2 - \bm{a}_3 = 0$

$\spalignmat{\vertbar{} \vertbar{} \vertbar{}; \bm{a}_1 \bm{a}_2 \bm{a}_3; \vertbar{} \vertbar{} \vertbar{} } \vector{4; 2; -1} = \vector{0; 0; 0}$

If we consider the SVD of a singular system as $\mathbf{A\,V} = \mathbf{U\,\Sigma}$ , then we can divide the $\bf{V}$ , $\bf{\Sigma}$ , and $\bf{U}$ matrices into blocks related to nonzero and zero valued singular values.

$\begin{aligned} &\mathbf{A}\,\mat{\tilde{\mathbf{V}} \mathbf{V}_{null}} = \mat{\tilde{\mathbf{U}} \mathbf{U}_{null}}\, \mat{\mathbf{\Sigma}; \mathbf{0}} \\ &\mathbf{A}\,\tilde{\mathbf{V}} = \tilde{\mathbf{U}}\,\mathbf{\Sigma}, \quad \text{and} \quad \mathbf{A}\,\mathbf{V}_{null} = \mathbf{U}_{null}\,\mathbf{0} = \mathbf{0} \end{aligned}$

The last $n - r$ columns of $\bf{V}$ is the null space because $\mathbf{A}\,\mathbf{V}_{null} = \mathbf{0}$ .

>> x = [4; 2; -1];
>> A*x
ans =
     0
     0
     0
>> x/norm(x)
ans =
    0.8729
    0.4364
   -0.2182
>> [~,~,V] = svd(A);
>> V(:,3) % = null(A)
ans =
    0.8729
    0.4364
   -0.2182

The null space of an under-determined matrix is at least ( $m - n$ ) columns. Remember from Under-determined Systems that under-determined systems have an infinite set of solutions. The RREF expresses the solution in terms of the general and particular solutions, while the null space from the SVD is the least squares solution. Although the solutions are different, they are both correct.

>> A = randi(20, 3, 5) - 8;
>> [~, S, V] = svd(A);
>> nullA = V(:, end-1:end);
>> A*nullA
ans =
   1.0e-14 *
   -0.0583    0.0992
    0.0444   -0.0666
   -0.0722   -0.1568

9.5.3. Row Space¶

The row space of a matrix $\bf{A}$ , which we denote as $\text{Row}( \bf{A} )$ , is the column space of $\mathbf{A}^T$ . It is the first $r$ columns of $\bf{V}$ from the SVD.

9.5.4. Left Null Space¶

The left null space of a matrix $\bf{A}$ are the set of row vectors, $\bf{Y}$ , that satisfy the relationship $\bf{Y\,A} = \bm{0}$ . It is also the right null space of $\mathbf{A}^T$ . The left null space of $\bf{A}$ may be found from the SVD as the transpose of the last ( $m - r$ ) columns of $\bf{U}$ . It is more common in MATLAB to compute it as the transpose of the right null space of $\mathbf{A}^T$ , Y = null(A’)’. The left null space exists for singular and over-determined matrices.

9.5.5. Orthogonality of Spaces¶

The four fundamental vector subspaces form interesting orthogonality relationships. Vectors in the null space are orthogonal to vectors in the row space. Similarly, vectors in the column space are orthogonal to vectors in the left null space.

We will use a $3{\times}5$ under-determined matrix to illustrate orthogonality between the null space and row space.

>> A = randi(15, 3, 5) - 6;
>> nullSpace = null(A);
>> size(nullSpace)
ans =
     5    3
>> rank(A)
ans =
    3

% The row Space is all columns of A^T.
>> rowSpace = A';

% Check for orthogonality with a dot product.
>> nullSpace'*rowSpace
ans =
   1.0e-14 *
         0         0         0
         0   -0.0444    0.1110

% The left null space is empty because the column space vectors
% are full rank.
>> left_null = null(A')
left_null =
    3x0 empty double matrix

Here is a $3{\times}3$ matrix of rank 1 to demonstrate orthogonality of the column space and the left null space.

>> a = randi(5, 3, 1);
>> a2 = a.*2;
>> a3 = a.*3;
>> A = [a a2 a3];
>> rank(A)
ans =
    1
>> colSpace = A(:,1);
>> left_null = null(A');
>> colSpace'*left_null
ans =
   1.0e-15 *
   -0.1943    0.4441

9.5. Fundamental Matrix Subspaces¶

9.5.1. Column Space¶

9.5.2. Null Space¶

9.5.3. Row Space¶

9.5.4. Left Null Space¶

9.5.5. Orthogonality of Spaces¶

Table of Contents

Previous topic

Next topic

This Page