Determinant of Matrices
The determinant of a square matrix $\mathbf{A}$ of size $n \times n$ is a mapping function from the space $\mathbf{R}^{n} \rightarrow \mathbf{R}$, denoted as $det(A)$.
$det(A)$ is used in various cases. For example, in 2-dimensional space, the absolute value of $det(A)$ is precisely the area of the shape formed by the column vectors in $A$. For instance, $A = \begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix}$. If you plot it in 2-dimensional space, you will see the shape formed is a rectangle with dimensions $3 \times 2$. The area of the shape is 6, which is also the result of $det(A) = (3 \times 2) - (0 \times 0)$.
In fact, the determinant of the matrix $A\ det(A)$ tells us how many times the size (volume) changes after being transformed by $A$ compared to the original size. In other words, consider the matrix $A$ as a mapping function $f$, let $X$ be the space to be transformed. When transforming $X$ by the mapping function $f$, we obtain the space $Y$, i.e., $f(X) = Y$. At this point, $det(A) = \frac{volume(Y)}{volume(X)}$.
Some properties:
- $\forall A \in \mathbf{R}^{n \times n}, |A| = |A^T|$
- $\forall A, B \in \mathbf{R}^{n \times n}, |AB| = |A||B|$
- $\forall A \in \mathbf{R}^{n \times n}, det(A) = 0$ if and only if the matrix $A$ is singular.
Eigenvalues and eigenvectors
For a matrix $\mathbf{A} \in \mathbf{R}^{n \times n}$, we call $\lambda \in \mathbf{C}$ and $v \in \mathbf{C}^n$ respectively an eigenvalue and eigenvector of $\mathbf{A}$ if:
$$ \begin{aligned} Av = \lambda v, v \neq 0 \end{aligned} $$
Before diving into finding $\lambda$ and $v$, let’s talk about why we need to find eigenvectors and eigenvalues, or in other words, the importance of these two variables. Look at the example below:
Linear transformation changes size
In geometric transformations (scale, shear, rotation, …), each transformation can be represented by a matrix $\mathbf{A}$. In this example, the transformation is represented by the matrix $A = \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix}$.
You can see in the 3 red, yellow, and blue vectors that 2 red and blue vectors remain unchanged in direction after being transformed by $\mathbf{A}$. These are exactly 2 eigenvectors of $\mathbf{A}$.
Eigenvectors are vectors that, after being transformed by $\mathbf{A}$, still lie within the span of the original vectors. They can change in direction, magnitude, or both, but they do not stray from their span. In the example above, the span of the green vector is the $Oy$ axis, and the span of the red vector is the $Ox$ axis. For the yellow vector, although not represented, its span is the line with the equation $x = y$. To understand easily, the span of the yellow vector is the line extended from the original yellow vector, in both directions up, down, and passing through the origin. After transformation, we see that the yellow vector is no longer within its span, so it is not an eigenvector of this transformation. The amount of change of each eigenvector corresponds to an eigenvalue of the transformation. For example, after transformation, the green vector is twice as long, corresponding to an eigenvalue of $\lambda = 2$. While the red vector does not change in magnitude, corresponding to $\lambda = 1$.
So, what’s the significance of finding eigenvalues and eigenvectors? Welp, this is a tough question that I often find poorly explained in math materials or websites. From my personal perspective after a period of study, I believe eigenvectors represent the essence of a linear transformation. They help you have a clear and understandable view of how a transformation is performed.
Finding eigenvalues and eigenvectors
Now we will prove the 2 eigenvalues $\lambda = 1$ and $\lambda = 2$ in the example above mathematically.
To find the eigenvalues $\lambda$ and eigenvectors $x$ of $\mathbf{A}$, always start with the expression:
$$ \begin{aligned} Av = \lambda v \end{aligned} $$
Why do we have this expression? The meaning of this expression is: multiplying matrix $A$ with an eigenvector $v$ yields a result similar to scaling eigenvector $v$ by some $\lambda$. This perfectly aligns with what we’ve learned earlier: after being transformed by $A$, the eigenvector $v$ only changes in direction or magnitude or both, but it remains within its span. $|\lambda|$ is the degree of change in magnitude of $v$ after the transformation, while if $\lambda < 0$, it means $v$ is reversed after transformation, and vice versa. What we need to do now is to find $v$ and $\lambda$ that satisfy this expression.
Transposing and changing signs, we have:
$$ \begin{aligned} Av - \lambda v &= 0\\ \Leftrightarrow (A - \lambda)v &= 0 \end{aligned} $$
Here, we’re back to the problem of finding the null space for the matrix $A - \lambda$. If you’re not familiar with what the null space is, please see here.
Remember, we have the constraint $v \neq 0$, and recall that the null space of $\mathbf{A}$ containing a vector other than the zero vector implies that $\mathbf{A}$ is non-singular. This is equivalent to $\mathbf{A}$ being a singular matrix - a singular matrix, equivalent to $det(A) = 0$.
So, the problem is reduced to finding conditions such that:
$$ \begin{aligned} det(A - \lambda) = 0 \end{aligned} $$
Applying to the matrix $A = \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix}$ in the example above, we have:
$$ \begin{aligned} \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix}v &= \lambda v\\ \Leftrightarrow \begin{bmatrix} 1 - \lambda & 0 \\ 0 & 2 - \lambda \end{bmatrix}v &= 0 \end{aligned} $$
To have $v \neq 0$, we must add the constraint:
$$ \begin{aligned} det(\begin{bmatrix} 1 - \lambda & 0 \\ 0 & 2 - \lambda \end{bmatrix}) &= 0 \\ \Leftrightarrow (1 - \lambda)(2 - \lambda) &= 0 \end{aligned} $$
From here, we obtain 2 solutions:
$$ \begin{aligned} \begin{cases} \lambda = 1 \\ \lambda = 2 \end{cases} \end{aligned} $$
So we have found 2 eigenvalues of the matrix $\mathbf{A}$, in line with what was said in the previous section. From these 2 eigenvalues, we can easily find $v$:
$$ \begin{aligned} \begin{bmatrix} 1 - \lambda & 0 \\ 0 & 2 - \lambda \end{bmatrix}v &= 0 \\ \Leftrightarrow \begin{cases} \begin{bmatrix} 0 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} &= 0 \\ \begin{bmatrix} -1 & 0 \\ 0 & 0 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} &= 0 \\ \end{cases} \end{aligned} $$
The result is that the matrix $\mathbf{A}$ has infinitely many eigenvectors, with each eigenvector having the form $\begin{bmatrix} x \\ 0 \end{bmatrix}$ or $\begin{bmatrix} 0 \\ y \end{bmatrix}$. We see in the original example that the green vector has the form $\begin{bmatrix} 0 \\ y \end{bmatrix}$ and the red vector has the form $\begin{bmatrix} x \\ 0 \end{bmatrix}$. Therefore, these are 2 eigenvectors of the transformation $A$.
Returning to finding eigenvalues and eigenvectors helps us understand the essence of the transformation. For $\lambda = 1$, we obtain $v$ as vectors of the form $\begin{bmatrix} x \\ 0 \end{bmatrix}$; and for $\lambda = 2$, $v$ are vectors satisfying $\begin{bmatrix} 0 \\ y \end{bmatrix}$. Looking back at the above illustration, the magnitude of the green vector after transformation by $A$ increases by a factor of 2, while the magnitude of the red vector remains unchanged, corresponding to $\lambda = 1$. Therefore, we understand that the matrix $A$ represents a transformation that increases the magnitude of vectors whose span is the $Oy$ axis by a factor of 2, while preserving the magnitude of vectors whose span is the $Ox$ axis. This is the essence of the transformation $A$.