In Part I, we established the essential language of linear algebra by introducing its foundational building blocks—scalars, vectors, matrices, and tensors—and the operations that connect them. We also learned how linear systems can be expressed in matrix form and how matrix rank reveals the structure of their solutions.
In this Part II, we move deeper into the core machinery of matrix algebra. We will introduce determinants, matrix inverses, eigenvalues, eigenvectors, and the Singular Value Decomposition (SVD). These concepts are essential for understanding how matrices transform space, how they encode important directions, and why they are central to modern computing, machine learning, and scientific applications.
Determinant
Every square matrix has an associated scalar value called its determinant, denoted by \(\det(A)\) or equivalently by \(|A|\). The determinant provides useful information about a matrix, including whether it is invertible. For a \(2 \times 2\) matrix;
\[ A= \begin{bmatrix} a & b\\ c & d \end{bmatrix}, \]the determinant is: \[\det(A)=ad-bc.\]
Although formulas also exist for calculating determinants of larger matrices, they quickly become
cumbersome. In practice, determinants of large matrices are computed using
numerical software—for example, R provides function det(), while
Python (NumPy) provides numpy.linalg.det() (often written as np.linalg.det()
after importing NumPy as np).
- The determinant is only defined for square matrices.
- Transposing a matrix does not change its determinant: \[ \det(A^{T})=\det(A). \]
- If a matrix has a zero row, a zero column, or two identical rows (or columns), then \[ \det(A)=0. \]
- The determinant of a product equals the product of the determinants: \[ \det(AB)=\det(A)\det(B). \]
- Multiplying a matrix by a scalar \(k\) multiplies its determinant by \(k^n\), where \(A\) is an \(n\times n\) matrix: \[ \det(kA)=k^n\det(A). \]
- A matrix is invertible if and only if \[ \det(A)\neq0. \] If \(\det(A)=0\), the matrix is called singular and has no inverse.
- If \(A\) is invertible, then \[ \det(A^{-1})=\frac{1}{\det(A)}. \]
Matrix Inverse
In Part I, we saw that a system of linear equations can be written compactly in matrix form as \(A\mathbf{x} = \mathbf{b}\). This notation allowed us to express the essential information of a linear system using the coefficient matrix \(A\), the variable vector \(\mathbf{x}\), and the constant vector \(\mathbf{b}\). In this part, we build on that foundation by introducing one of the most important concepts in matrix algebra: the matrix inverse.
Unlike ordinary arithmetic, matrix algebra does not have a division operation. Instead, we use an operation called "multiplying by an inverse". To understand this, recall a familiar fact from basic algebra: every nonzero number has a multiplicative inverse. For example, the inverse of a scalar \(n \) is \(1/n\). Inverses have the property that a number multiplied by its inverse gives \(1\):
\[ n\left(\frac{1}{n}\right)=1. \]The same is true in matrix math! The matrix equivalent of \(1/n\) is the inverse matrix, denoted by \(A^{-1}\). It has the property that multiplying a matrix by its inverse gives the identity matrix:
\[ AA^{-1}=A^{-1}A=I. \]where identity matrix \(I\) plays the same role in matrix multiplication as the number \(1\) does in ordinary multiplication. Multiplying any matrix by \(I\) leaves it unchanged.
Matrix inverses are particularly useful for solving systems of linear equations. Given the matrix equation: \(A\mathbf{x}=\mathbf{b}\); if \(A\) has an inverse, we can multiply both sides by \(A^{-1}\); since \(A^{-1}A=I\), this simplifies to:
\[ A^{-1}A\mathbf{x}=A^{-1}\mathbf{b}, \] \[ \mathbf{x}=A^{-1}\mathbf{b}. \]Systems of linear equations are not just a mathematical abstraction—they appear everywhere in applied science. Numerical simulations in physics and engineering routinely require solving enormous linear systems, quantitative finance uses them when discretizing and solving the Black-Scholes PDE for options pricing, and reinforcement learning relies on solving the Bellman equation, which is itself a system of linear equations. This makes efficient methods for solving \(A\mathbf{x}=\mathbf{b}\) essential in modern computation.
So, how do we actually compute a matrix inverse? There are several methods used to compute the inverse matrix, when it exists. Two classical hand-calculation methods are commonly taught:
1. Adjugate (cofactor) method:
If \(A\) has a nonzero determinant, its inverse can be written as:
\[ A^{-1} = \frac{1}{\det(A)}\operatorname{adj}(A), \]where \(\det(A)\) is the determinant of \(A\), and \(\operatorname{adj}(A)\) is the adjugate matrix, formed from the cofactors of \(A\).
A cofactor is obtained by temporarily removing the row and column containing a particular element, computing the determinant of the remaining submatrix (called its minor), and applying a positive or negative sign according to the element's position. Collecting all of these cofactors into a matrix (and then transposing it) produces the adjugate matrix.
2. Gauss-Jordan elimination method:
Gauss-Jordan elimination is an algorithmic procedure wereby, instead of computing determinants and cofactors; it augments the matrix with the identity matrix and performs row operations until the left side becomes \(I\). The right side then becomes \(A^{-1}\).
Although these methods are mathematically elegant, they become computationally expensive for large matrices. In practice,
numerical software such as R's solve() or Python's numpy.linalg.solve() solves
\(A\mathbf{x} = \mathbf{b}\) directly without explicitly computing \(A^{-1}\). When an explicit inverse is required,
functions such as numpy.linalg.inv() are available. These numerical software use efficient matrix factorization algorithms such as
LU decomposition, QR decomposition, Cholesky decomposition, or the
Singular Value Decomposition (SVD).
- Not every matrix has an inverse. A matrix that does is called invertible (or nonsingular). Matrices without an inverse are called singular.
-
A special class of matrices, called orthogonal matrices, have
orthonormal columns. This means two things:
- Each column has unit length (length \(= 1\)). The length of a column vector is found by squaring each element, adding the squares, and taking the square root. Every column must have length 1.
- Different columns are orthogonal. This means the dot product of any two distinct columns is 0. Geometrically, the columns meet at right angles (90°), so they point in completely independent directions (Mutually perpendicular).
- The identity matrix mentioned above is also another class of matrices. It is a special type of diagonal matrix, meaning that all entries off the main diagonal are zero. For the identity matrix, every entry on the main diagonal is equal to 1. For example, \[ I_3= \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{bmatrix} \]
-
In general, the columns of a matrix are not orthonormal by default. However, if the columns are
linearly independent, they can be converted into an orthonormal set using
the Gram-Schmidt process. This procedure constructs new orthonormal vectors
from the original set while ensuring that the new vectors still describe the same space as the original ones.
The Gram-Schmidt process is one of the fundamental algorithms used to compute the QR decomposition.
A detailed treatment of the algorithm is beyond the scope of this blog, so it is mentioned
here only as a point of reference for readers interested in exploring the topic further.
Eigenvalues and Eigenvectors
So far, we have seen that matrices can be added, multiplied, inverted, and used to solve systems of equations. Another important question is: does a matrix have any special directions?
Imagine applying a matrix transformation to a vector. Most vectors will change both their length and their direction. However, some special vectors behave differently: after the transformation, they point in exactly the same direction. These special vectors are called eigenvectors. An eigenvector is a nonzero vector whose direction remains unchanged when multiplied by a square matrix. The matrix only stretches, shrinks, or reverses the vector. The amount of scaling is called the eigenvalue. In general:
\[ A\mathbf{x}=\lambda\mathbf{x}, \]where \(\mathbf{x}\) is an eigenvector and \(\lambda\) is its corresponding eigenvalue. If \(\lambda > 1\), the vector is stretched. If \(0 < \lambda < 1\), it is compressed. If \(\lambda\) is negative, the vector reverses direction. The important point is that the direction of the eigenvector remains unchanged.
Finding Eigenvalues and Eigenvectors
Eigenvalues are found by solving the characteristic equation:
\[ \det(A-\lambda I)=0, \]where \(I\) is the identity matrix. Solving this equation gives the possible eigenvalues of the matrix. Once an eigenvalue is known, the corresponding eigenvector can be found by solving
\[ (A-\lambda I)\mathbf{x}=0. \]Each eigenvalue may have one or more associated eigenvectors. For small matrices, these calculations can often be performed by hand. For larger matrices, however, solving the characteristic equation becomes impractical. Instead, numerical algorithms are used to estimate eigenvalues and eigenvectors efficiently.
One of the simplest is the power method, which is an iterative algorithm that repeatedly multiplies a vector by the matrix until it converges to the eigenvector associated with the largest (dominant) eigenvalue. Modern software such as R, Python (NumPy/SciPy), and MATLAB uses more sophisticated numerical algorithms to compute eigenvalues accurately and efficiently.
Eigenvalue Decomposition
Not every square matrix has enough linearly independent eigenvectors. When it does, the matrix is said to be diagonalizable and it can be written in a special form called eigenvalue decomposition:
\[ A=Q\Lambda Q^{-1}, \]where:
- \(Q\) contains the eigenvectors of \(A\) as its columns.
- \(\Lambda\) is a diagonal matrix containing the corresponding eigenvalues.
- \(Q^{-1}\) inverses the transformation created by \(Q\); transforming the coordinates back to the original basis.
Eigenvalue decomposition separates a complicated matrix transformation into three simpler steps: first transforming into the coordinate system defined by the eigenvectors, then independently scaling each of those directions according to its eigenvalue, and finally transforming back to the original coordinate system.
Singular Value Decomposition (SVD)
Eigenvalue decomposition is extremely useful, but it has an important limitation: it only applies to certain square matrices. A more general decomposition called the Singular Value Decomposition (SVD) works for every matrix, including rectangular matrices. In general:
\[ A=U\Sigma V^{T}, \]Unlike eigenvalue decomposition, the SVD is constructed from two related eigenvalue problems. The columns of \(V\) are the eigenvectors of \(A^{T}A\), while the columns of \(U\) are the eigenvectors of \(AA^{T}\). The diagonal entries of \(\Sigma\), called singular values, are the square roots of the eigenvalues of these matrices. In this way, SVD extends the ideas of eigenvalues and eigenvectors to every matrix.
- If \(A\) is an \(m \times n\) matrix with rank \(r\), then:
- \( A^{T}A \) is an \(n \times n\) matrix and therefore has \(n\) eigenvalues.
- Likewise, \( AA^{T} \) is an \(m \times m\) matrix and therefore has \(m\) eigenvalues.
- Although these matrices may have different sizes, they have exactly the same \(r\) positive eigenvalues.
- Any additional eigenvalues are zero, reflecting the difference in dimensions.
- The singular values of \(A\) are the square roots of these shared eigenvalues: \[ \sigma_i=\sqrt{\lambda_i}, \qquad i=1,\ldots,r. \]
SVD's three matrices have distinct roles:
- \(V\) contains the right singular vectors, which describe the most important directions in the input space.
- \(\Sigma\) is a diagonal matrix containing the singular values, which measure how much each of these directions is stretched or compressed.
- \(U\) contains the left singular vectors, which describe the corresponding directions in the output space after the transformation.
Geometrically, SVD decomposes every matrix transformation into three simple operations:
- Rotate (or change coordinates) into the principal input directions using \(V^{T}\).
- Stretch or compress independently along those directions using \(\Sigma\).
- Rotate into the final output coordinate system using \(U\).
Large singular values correspond to directions that preserve most of the information in the data, while very small singular values often represent noise or redundant information. This property makes SVD particularly useful for dimensionality reduction, data compression, and identifying the most important patterns in large datasets.
Because SVD works for every matrix and is numerically stable, it is one of the most important tools in modern statistics, machine learning, and data science. It forms the computational foundation of techniques such as Principal Component Analysis (PCA), recommendation systems, image compression, natural language processing, and many optimization algorithms.
- Positive definite matrices have all positive eigenvalues, while positive semidefinite matrices have eigenvalues that are all positive or zero. Likewise, negative definite and negative semidefinite matrices have all negative, or negative-or-zero, eigenvalues respectively.
- Positive definite matrices arise naturally in optimization, covariance matrices, and machine learning. They guarantee unique solutions in many problems and enable efficient algorithms such as the Cholesky decomposition.
Conclusion
Linear algebra is an incredibly broad field, with entire textbooks devoted exclusively to its theory and applications. Parts I and II of this series were never intended to cover every aspect of the subject. Instead, their purpose has been to establish a solid foundation by introducing the fundamental concepts, ideas, and terminology that underpin linear algebra. With this foundation in place, readers should now have the language and intuition needed to explore the subject with confidence.
Now that we have established the core theory of linear algebra, Parts III and IV will shift the focus from mathematical foundations to practical applications. We will explore how the concepts introduced throughout these first two parts form the backbone of modern technologies, powering everything from computer graphics and scientific computing to machine learning and artificial intelligence. By connecting the theory to real-world problems, we will see why linear algebra is one of the most important mathematical tools in science and engineering.