Four fundamental subspaces in Linear Algebra - What are they?
- Row space ($C(A^{T})$), a subspace of $\mathbf{R}^n$
- Column space ($C(A)$), a subspace of $\mathbf{R}^m$
- Nullspace ($N(A)$), a subspace of $\mathbf{R}^n$
- Left nullspace ($N(A^T)$), a subspace of $\mathbf{R}^m$
We have extensively covered the column space and nullspace.
The row space contains all linear combinations of the rows in a matrix.
But what if we don’t like dealing with rows and only want to work with column vectors? Then you can transpose the matrix $A$ to $A^T$, which means the rows in $A$ are now the columns in $A^T$. Hence, it can be stated that the row space of matrix $A$ is the same as the column space of $A^T$. Both of these spaces can be denoted using the symbol $C$.
Similarly, the left nullspace of matrix $A$ is the nullspace of matrix $A^T$.
To find the left nullspace of $A$, we solve the equation $A^Ty = 0$. It’s called the left nullspace because, when we transpose both sides of the equation, we have:
$$ \begin{aligned} (A^{T}y)^{T} & = 0^{T} \\ \Leftrightarrow y^{T}A^{TT} & = 0^{T} \\ \Leftrightarrow y^{T}A & = 0^{T} \end{aligned} $$
We see that now $y$ is on the left side, $y^T$ and $0^T$ are row vectors.
Dimensions of the Four subspaces
Let’s determine the basis and dimension of the four vector spaces mentioned above. Recall that the dimension of a vector space is the number of vectors in any basis of that space. A basis of a vector space is a set of vectors with two properties: linear independence and spanning a vector space.
Column Space
For a matrix $A$ of size $m \times n$.
The dimension of the column space of $A\ dim(C(A)) = r$. The pivot columns in $A$ form the basis of $C(A)$.
Since the rank of a matrix $r(A)$ is the number of pivot columns of matrix $A$. Pivot columns indicate they are linearly independent vectors in matrix $A$. Furthermore, the columns of $A$ span the column space $C(A)$.
Therefore, the set of pivot columns of matrix $A$ is the basis of the column space $C(A)$.
And the number of vectors in the basis is the dimension of the vector space, so $dim(C(A)) = r$.
Consider the example: $A = \begin{bmatrix} 1 & 3 & 5 & 0 & 7 \\ 0 & 0 & 0 & 1 & 2\\ 1& 3 & 5&1 &9 \end{bmatrix}$
Applying row operations to reduce it to row-echelon form, we have:
$$ \begin{aligned} A & = \begin{bmatrix} 1 & 3 & 5 & 0 & 7 \\ 0 & 0 & 0 & 1 & 2\\ 1& 3 & 5&1 &9 \end{bmatrix} \rightarrow R & = \begin{bmatrix} 1 & 3 & 5 & 0 & 7 \\ 0 & 0 & 0 & 1 & 2\\ 0 & 0 & 0 & 0 & 0 \end{bmatrix} \end{aligned} $$
We see that $r(A) = r(R) = 2$.
Note: $C(A) \neq C(R)$!. The reason is that during the transformation from $A$ to $R$, we performed row operations. These operations ensure the preservation of the row space but not the column space.
However, the number of vectors $dim(C(A)) = dim(C(R))$. Columns 1 and 4 of $R$ form the basis of $C(R)$; similarly, columns 1 and 4 of $A$ form the basis of $C(A)$.
Row Space
A wonderful fact: the dimension of the row space $dim(C(A^T)) = dim(C(A)) = r$.
Taking the matrix $A$ from the previous example, to find the rank of $A^T$, we first need to transpose $A$ to $A^T$:
$$ \begin{aligned} A & = \begin{bmatrix} 1 & 3 & 5 & 0 & 7 \\ 0 & 0 & 0 & 1 & 2\\ 1& 3 & 5&1 &9 \end{bmatrix} \rightarrow A^T & = \begin{bmatrix} 1 & 0 & 1 \\ 3 & 0 & 3 \\ 5 & 0 & 5 \\ 0 & 1 & 1 \\ 7& 2 &9 \end{bmatrix} \rightarrow R^T & = \begin{bmatrix} 1 & 0 & 1 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 1 & 1 \\ 0 & 0 & 0 \end{bmatrix} \rightarrow R & = \begin{bmatrix} 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 \\ 1 & 0 & 0 & 1 & 0 \end{bmatrix} \end{aligned} $$
We have $R^T$ (and $A^T$) with 2 pivot columns, which are column 1 and column 2.
It can be observed that the rows in $R$ span the row space $C(R^T)$. Similarly, the rows in $A$ span the row space $C(A^T)$. However, row 3 is a linear combination of rows 1 and 2, so in this case, the basis of $C(A^T)$ is the first 2 rows of $A$.
The basis of the row space $C(A^T)$ is the first $r$ rows of $A$
From the above examples, we can deduce $dim(C(A^T)) = dim(C(A)) = r(A) = 2$.
Nullspace
The dimension of the nullspace of $A$ $dim(N(A)) = n - r$. The solutions to the equation $Ax = 0$ form the basis of $N(A)$.
When solving the equation $Ax = 0$, for each free column $A_i$ of $A$, corresponding to the free variable $x_i$, we will find a solution. Thus, the number of solutions to the main system equals the number of free columns of $A$, which is $n - r$, where $r$ is the number of pivot columns (also the rank of $A$).
Since the solutions to the system $Ax = 0$ are linearly independent, they form the basis for the null space $N(A)$.
Left Nullspace
To find the left nullspace of $A$, we solve the equation $A^Ty = 0$ or $y^TA = 0^T$ or $R^Ty = 0$ and $y^TR = 0^T$ with $R = rref(A)$.
The basis of the left nullspace of $A$ is the first $r$ rows of $A$
Continuing with $A = \begin{bmatrix} 1 & 3 & 5 & 0 & 7 \\ 0 & 0 & 0 & 1 & 2\\ 1& 3 & 5&1 &9 \end{bmatrix}$ as an example, I will solve the system $y^{T}R = 0^{T}$:
$$ \begin{aligned} y^{T}R = 0^{T} &\Leftrightarrow \begin{bmatrix} y_1 & y_2 & y_3 \end{bmatrix} \begin{bmatrix} 1 & 3 & 5 & 0 & 7 \\ 0 & 0 & 0 & 1 & 2 \\ 0 & 0 & 0 & 0 & 0 \end{bmatrix} = \begin{bmatrix} 0 & 0 & 0 & 0 & 0 \end{bmatrix} \\ \end{aligned} $$
The system above can be rewritten as:
$$ \begin{aligned} & y_1\begin{bmatrix}1 & 3 & 5 & 0 & 7 \end{bmatrix} \\ + & y_2\begin{bmatrix}0 & 0 & 0 & 1 & 2\end{bmatrix} \\ + & y_3\begin{bmatrix}0 & 0 & 0 & 0 & 0\end{bmatrix} \\ = & \begin{bmatrix}0 & 0 & 0 & 0 & 0\end{bmatrix} \end{aligned} $$
It is easy to see that to satisfy the system above, we only need $y_1 = y_2 = 0$ and $y_3$ can be any value.
The left nullspace of $R^T$ contains all sets of vectors $y = (0, 0, y_3)$.
In a matrix $R$ with rank $r$ and size $m \times n$, there will always be $m - r$ rows of 0. Any linear combination of $m - r$ 0-rows will always give the vector $0$. Only a single linear combination of all rows in $R$ will give the vector $0$. Thus, $y$ in the left nullspace is the vector with $y_1 = y_2 = … = y_r = 0$.
If $A$ is a matrix of size $m \times n$ and rank $r$, the left nullspace of $A$ has dimension $dim(N(A^T)) = m - r$.
The Big Picture
To have an overview of this article, take a look at the image below:
The overview of the four fundamental subspaces in linear algebra
Here are some properties you need to remember:
1. Matrices $A$ and $R$ have the same row space. These spaces have the same dimension $r$ and the same basis.
The reason is that as mentioned above, the transformation from $A$ to $R$ through row operations preserves the row space but changes the column space.
2. In matrix $A$, the number of linearly independent column vectors equals the number of linearly independent row vectors.
This means that we have $dim(C(A)) = dim(C(A^T)) = r$.
3. The nullspace of $A$ and $R$ are the same, with the same dimension $n-r$ and the same basis.
This is because the transformations $A \rightarrow R$ do not change the solutions. For each free variable $x_i$ in the system $Ax = 0$ or $Rx = 0$, we can find a solution. Thus, $N(A) = N(R) = n - r$.
4. The left nullspace of $A$ (the nullspace of $A^T$) has dimension $m-r$. We will use the counting theorem to prove this.
First, the column space of $A^T$ is the same as the row space of $A$. According to property 2, we have $dim(C(A)) = dim(C(A^T)) = r$. Moreover, $A^T$ now has size $n \times m$, meaning its column vectors lie in $\mathbf{R}^m$.
From the above and using the counting theorem, we will have: $r + x = m$, or in other words, $x = m-r$. Thus, the left nullspace of $A^T$ (also the nullspace of $A$) has dimension $dim(C(A^T)) = m-r$.