Linear Independence and Linear Dependence
A set is called linearly independent when the vectors in that set are linearly independent of each other.
In other words, we cannot represent a vector as a linear combination of any other vectors in the set. Otherwise, the set is linearly dependent.
For example: given a set of 3 vectors $v_1 = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix} \quad ; \quad v_2 = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix} \quad ; \quad v_3 = \begin{bmatrix} 1 \\ 0 \\ 2 \end{bmatrix}$
This set is not linearly independent because vector $v_3 = v_1 + v_2$. Note that a set containing the zero vector is certainly linearly dependent because the zero vector can be written as a linear combination of any vectors $v$ in the set.
For any matrix $A$, we can check if the vectors in $A$ are linearly independent by solving the system of equations $A\mathbb{x} = \mathbb{0}$. Why can we determine this through this method? Recall what we learned in the post about nullspace. In that post, we learned that when a matrix $A$ has all columns containing pivot elements, then $A\mathbb{x} = \mathbb{0}$ has only the trivial solution $x = \mathbb{0}$. And when all columns of $A$ contain pivot elements, it means that no column of $A$ is a linear combination of the other columns $\Leftrightarrow$ $A$ is linearly independent.
The column vectors of $A$ are linearly independent when the system $A\mathbb{x} = \mathbb{0}$ has only the trivial solution $x = \mathbb{0}$.
In other words, the vector set $v_1, v_2, …, v_n$ is linearly independent if and only if $x_1v_1 + x_2v_2 + … + x_nv_n = \mathbb{0}$ holds only with $x_1 = x_2 = … = x_n = \mathbb{0}$.
Another note: a set of 3 vectors belonging to the space $\mathbf{R}^2$ is always linearly dependent.
Assuming a matrix $A$ of size $2 \times 3$, then the number of pivot elements $r$ will always be $\leq 2$. Therefore, we always have at least 1 free variable and $A\mathbb{x} = \mathbb{0}$ will always have a non-zero solution.
Any set of $n$ vectors $\in \mathbf{R}^m$ are linearly dependent if $m < n$.
When the number of pivot elements $r = $ the number of columns of the matrix $n$, then the matrix has full column rank. In the post about solving systems $A\mathbb{x} = b$, we learned that full column rank indicates that the matrix is linearly independent.
The columns of $A$ are linearly independent when $r = n \Leftrightarrow $ the system of equations $A\mathbb{x} = \mathbb{0}$ has no free variables
Span
A set of vectors $S$ is the span of the vector space $V$ if every vector in $V$ can be represented as a linear combination of vectors in $S$.
This means if every vector $v \in V$ can be written as a linear combination of vectors $s \in S$, then we say $S$ spans $V$.
Additionally, we have other terms like: $S$ spans $V$, $V$ is spanned by $S$, $V$ is the linear span of $S$.
Let’s talk about the column space $C(A)$ of a matrix $A$.
$C(A)$ contains all vectors $b$ that are the result of the system of equations $A\mathbb{x} = b$.
Therefore, these $b$ are being represented as linear combinations of columns in $A$, equivalent to the set of column vectors of $A$ spanning the column space of $A$.
For example: the set of vectors $v_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$ and $v_2 = \begin{bmatrix} 0 \\ 1 \end{bmatrix}$ spans the 2-dimensional space $\mathbf{R}^2$.
It’s evident because any vector $u \in \mathbf{R}^2$ can be written as $u = cv_1 + wv_2$.
However, the set of vectors $w_1 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$ and $v_2 = \begin{bmatrix} -1 \\ -1 \end{bmatrix}$ only spans a line in $\mathbf{R}^2$.
Basis and Dimension
A basis of a vector space is a set of vectors $v_1, v_2, …, v_n$ that satisfy 2 properties:
- $v_1, v_2, …, v_n$ are linearly independent
- $v_1, v_2, …, v_n$ span a vector space
This means if the vectors in the spanning set $S$ are linearly independent, then $S$ is the basis of a vector space generated by $S$.
A basis is a set with just enough vectors to span a space.
As in the example above, we see that the set $v_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$ and $v_2 = \begin{bmatrix} 0 \\ 1 \end{bmatrix}$ spans $\mathbf{R}^2$.
However, not every set of 2 independent vectors is a basis for $\mathbf{R}^2$ ($w_1 = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$ and $v_2 = \begin{bmatrix} -1 \\ -1 \end{bmatrix}$ is an example).
Similarly, not every set of 3 independent vectors is a basis for $\mathbf{R}^3$. Sometimes they only span a line or a 2-dimensional plane.
The vectors $v_1, v_2, …, v_n$ form the basis of the space $\mathbf{R}^n$ if and only if they are the columns of an invertible matrix of size $n \times n$
Hence, the basis of $\mathbf{R}^n$ is an infinite set.
For example, the columns of matrix $A = \begin{bmatrix} 1 & 0 & 0 \\ 1 & 1 & 0 \\ 1 & 0 & 0 \end{bmatrix}$ form a basis of the space $\mathbf{R}^3$.
The columns of matrix $B = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$ also form a basis of the space. This is called the standard basis of $\mathbf{R}^3$.
Moreover, with $v_1, v_2, …, v_n$ being vectors in the basis of $\mathbf{R}^n$, the linear combination to create a vector $v$ is unique.
It means, if $v = a_1v_1 + a_2v_2 + … + a_nv_n$ and $v = b_1v_1 + b_2v_2 + … + b_nv_n$, then $a_i = b_i$ for $1 \leq i \leq n$.
Indeed, subtracting the two vectors $v$ yields $v - v = (a_1 - b_1)v_1 + (a_2 - b_2)v_2 + … + (a_n - b_n)v_n = \mathbb{0}$. Since $v_1, v_2, …, v_n$ are linearly independent, $a_i - b_i = \mathbb{0} \Leftrightarrow a_i = b_i$.
A question I had when starting this chapter was: can there exist a basis for $\mathbf{R}^n$ with a number of vectors different from $n$?
The answer is: NO. Every basis of a vector space must have the same number of vectors.
This leads to a concept of the dimension of a vector space:
The number of vectors in any and every basis of a vector space is the dimension of that space
This means if $v_1, v_2, …, v_n$ and $w_1, w_2, …, w_m$ are both bases of a space, then $n = m$.
The number of vectors in a basis depends on the dimension of the space. The space $\mathbf{R}^n$ has $n$ vectors in any basis of it.