Linear algebra is one of the most widely used branches of mathematics in science, engineering, and artificial intelligence (AI); yet it is often taught in a way that feels abstract and disconnected from real-world application. Most introductory courses in linear algebra stop at definitions and some hand-solved exercises. As a result, many students walk away knowing how to compute a determinant or eigenvalue but not knowing why these ideas matter or where they are used in real-world applications.

In reality, linear algebra is everywhere. It powers modern technologies such as deep learning, computer graphics, robotics, computer vision, signal processing, financial modelling, and even the way our phones processes images. Of course, this is just a fraction of what you can do with linear algebra; there are many other uses like ranking in search engines, cryptography and data security, bioinformatics, recommendation systems, amongst many other applications.

This blog is the first in a two-part series designed to bridge the gap between mathematical theory and real-world application, which was my main goal when I launched this site. In this first part, I will introduce the essential terminology and ideas that help in understanding the language of linear algebra. Then in the second part I will demonstrate how these ideas are applied in different domains with three solid examples. If you are already comfortable with the fundamental concepts and prefer to explore applications, feel free to jump directly to Part II.

Why Learn Linear Algebra?

Linear algebra is one of the three main mathematical “languages” of AI (alongside calculus and probability). It provides a framework for representing and manipulating data efficiently. In fact, modern computers are designed to perform millions of matrix operations every second, making linear algebra one of the foundations of scientific computing.

Many popular Python libraries—including NumPy, SciPy, Pandas, TensorFlow, and PyTorch—are built almost entirely around vector and matrix operations. Even the simplest machine learning model, linear regression, relies heavily on linear algebra. This means understanding linear algebra is essential, especially if you want to thrive in the modern technological age.

Linear Algebra Building Blocks

1. Scalars

A scalar is simply a single numerical value. Each element (numbers in a matrix or vector) is a scalar. Think of a scalar as a single measurement, such as: Temperature = 24°C or Salary = \$6,600.

2. Vectors

A vector is an ordered collection of numbers. Vectors are commonly written as either row vectors (horizontal) or column vectors (vertical). For example, suppose a shop sells 15 laptops, 22 phones, and 10 tablets. These sales can be represented by the column vector: \[ \mathbf{s} = \begin{bmatrix} 15 \\ 22 \\ 10 \end{bmatrix} \]

3. Matrices

Put simply, a matrix is an array of numbers organised into rows and columns. Matrices provide a convenient way to organise and store data; in fact, many real-world datasets can naturally be represented as matrices. For example, suppose we record the daily closing prices of two stocks over three consecutive days. This information can be organised into a \(2 \times 3\) matrix, where the two rows represent the stocks and the three columns represent the days: \[ P = \begin{bmatrix} p_{1,1} & p_{1,2} & p_{1,3} \\ p_{2,1} & p_{2,2} & p_{2,3} \end{bmatrix} \]

Matrices are widely used to represent pixels in images, spreadsheets, databases, neural network weights, etc. Matrices can either be square (\(n \times n\)) or rectangular (\(m \times n\)).

4. Tensors

A tensor is a generalisation of a scalar, vector, and matrix to higher dimensions. Scalars are 0th-order tensors, vectors are 1st-order tensors, and matrices are 2nd-order tensors. Higher-order tensors are particularly useful for representing data with three or more dimensions. For example, a colour image can be represented as a third-order tensor with dimensions \((\text{height} \times \text{width} \times \text{RGB channels})\). The elements of a third-order tensor are typically written as \(a_{i,j,k}\), while the elements of a fourth-order tensor are written as \(a_{i,j,k,l}\), and so on. Modern machine learning frameworks, such as TensorFlow, are named after tensors because tensors are the primary data structures used to store and manipulate data during computation.

Matrix Addition/Subtraction

Two matrices can only be added or subtracted if they have the same dimensions, which means they have the same number of rows and columns. The operation is performed element by element. For example;

\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 1+5 & 2+6 \\ 3+7 & 4+8 \end{bmatrix} = \begin{bmatrix} 6 & 8 \\ 10 & 12 \end{bmatrix} \]

Matrix subtraction works in exactly the same way, except corresponding elements are subtracted instead of added.

Matrix Multiplication

Matrix multiplication is one of the most important operations in linear algebra. It combines the rows of one matrix with the columns of another. Unlike matrix addition, the two matrices do not have to be the same shape. However, the number of columns of the first matrix has to equal the number of rows of the second matrix. For example; if \(A\) is an \(n \times m\) matrix and \(B\) is an \(m \times p\) matrix, then the product \(AB\) is defined and has dimensions \(n \times p\):

\[ (n \times m)(m \times p) = n \times p. \]

Notice that the number of columns in A \(= \) number of rows in B; which is (\(m \)). Each element is obtained by multiplying a row of \(A\) by a column of \(B\):

\[ AB_{ij}=\sum_{k}A_{ik}B_{kj}. \]

For example,

\[ \begin{bmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \end{bmatrix} \begin{bmatrix} 7\\ 8\\ 9 \end{bmatrix} = \begin{bmatrix} 1(7)+2(8)+3(9)\\ 4(7)+5(8)+6(9) \end{bmatrix} = \begin{bmatrix} 50\\ 122 \end{bmatrix} \]

Here, each entry of the result is the dot product of a row of the first matrix and a column of the second. For larger matrices the process is exactly the same, but repeated for every entry of the output matrix. Modern software performs these calculations efficiently for matrices with millions of elements.

Matrix Transpose

The transpose of a matrix is obtained by swapping its rows and columns, i.e. the rows become columns. If \[ A = \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix}, \] then its transpose is \[ A^{T} = \begin{bmatrix} a_{11} & a_{21} \\ a_{12} & a_{22} \end{bmatrix} \]

So, why do we need it? Suppose a supermarket records sales of three products over four days. Rows represent products, columns represent days. Now imagine you want to compare sales across days instead of products. Rather than rewriting the entire dataset, we simply transpose it. The transpose allows us to switch perspectives without changing the underlying information.

Determinant

Every square matrix has an associated scalar value called its determinant, denoted by \(\det(A)\) or equivalently by \(|A|\). The determinant provides useful information about a matrix, including whether it is invertible. For a \(2 \times 2\) matrix;

\[ A= \begin{bmatrix} a & b\\ c & d \end{bmatrix}, \]

the determinant is: \[\det(A)=ad-bc.\]

Although formulas also exist for larger matrices, they quickly become cumbersome. In practice, determinants of large matrices are computed using numerical software—for example, R provides function det(), while Python (NumPy) provides numpy.linalg.det() (often written as np.linalg.det() after importing NumPy as np).

Matrix Inverse

Unlike ordinary arithmetic, matrix algebra does not have a division operation. Instead, we use an operation called "multiplying by an inverse". To understand this, recall a familiar fact from basic algebra: Every nonzero number has a multiplicative inverse. For example; the inverse of a scalar \(n \), is \(1/n\). Inverses have the property that a number multiplied by its inverse \(= 1\):

\[ n\left(\frac{1}{n}\right)=1. \]

The same is true in matrix math! The matrix equivalent of \(1/n\) is the inverse matrix, denoted by \(A^{-1}\). It has the property that multiplying a matrix by its inverse gives the identity matrix:

\[ AA^{-1}=A^{-1}A=I. \]

The identity matrix \(I\) plays the same role in matrix multiplication as the number \(1\) does in ordinary multiplication. Multiplying any matrix by \(I\) leaves it unchanged.

Matrix inverses are particularly useful for solving systems of linear equations. Given the matrix equation: \(A\mathbf{x}=\mathbf{b}\); if \(A\) has an inverse, we can multiply both sides by \(A^{-1}\); since \(A^{-1}A=I\), this simplifies to:

\[ A^{-1}A\mathbf{x}=A^{-1}\mathbf{b}, \] \[ \mathbf{x}=A^{-1}\mathbf{b}. \]

So, how do we actually compute a matrix inverse? There are many methods used to compute the inverse matrix, when it exists. Two methods are typically used for hand calculations. The first one is Adjugate (cofactor) method; if \(A\) has a nonzero determinant, then its inverse can be written as: \[ A^{-1}=\frac{1}{\det(A)}\operatorname{adj}(A), \] where \(\det(A)\) is the determinant of \(A\), and \(\operatorname{adj}(A)\) is the adjugate matrix, formed from the cofactors of \(A\). (A cofactor is obtained by temporarily removing the row and column containing a particular element, computing the determinant of the remaining submatrix (called its minor), and applying a positive or negative sign according to the element's position. Collecting all of these cofactors into a matrix (and then transposing it) produces the adjugate matrix). Another method is the Gauss-Jordan elimination, which is an algorithmic procedure wereby instead of computing determinants and cofactors, you augment the matrix with the identity matrix.

Although these direct formulas are mathematically elegant, for larger matrices they become computationally expensive, because the calculations quickly becomes tedious as the size of the matrix increases. In practice, numerical software such as R's solve() or Python's numpy.linalg.solve() is usually used to solve \(A\mathbf{x}=\mathbf{b}\) directly, avoiding the need to explicitly compute \(A^{-1}\). When an explicit inverse is required, functions such as numpy.linalg.inv() are available. These numerical software uses efficient matrix factorization algorithms such as LU decomposition, QR decomposition, Cholesky decomposition , or the Singular Value Decomposition (SVD)

Matrix Rank

The rank of a matrix is the number of linearly independent rows (or columns) it contains. A row is linearly independent if it cannot be written as a linear combination of the other rows. Intuitively, the rank measures how much unique information is contained in a matrix.

Matrix rank plays an important role in statistics and machine learning. It is particularly useful when solving systems of linear equations, because it tells us how many of the equations provide independent information. Dependent equations do not add any new information and therefore do not increase the rank.

Example 1: A Unique Solution

Consider the system:

\[ \begin{aligned} y &= 2x+1\\ y &= -x+4 \end{aligned} \]

One way to solve this system is by graphing the two lines. The solution is the point where they intersect. Since the equations are linearly independent, the coefficient matrix has rank 2, giving a unique solution \((1,3)\).

When a system has two variables, each equation represents a line, and the solution is their point of intersection. With three variables, each equation represents a plane in 3-dimensional space, and the solution is the point where all three planes intersect. While these geometric interpretations are very helpful, they quickly become impossible to visualize in higher dimensions. Fortunately, software such as R and Python can solve these larger systems efficiently.

Example 2: Infinitely Many Solutions

Now consider the system:

\[ \begin{aligned} x+y&=2\\ 2x+2y&=4 \end{aligned} \]

The second equation is simply twice the first, so it does not provide any new information. The two equations therefore are linearly dependent. The coefficient matrix has rank 1, because there is only one independent equation. When plotted they represent the same line and every point on the line satisfies both equations, so the system has infinitely many solutions.

Example 3: No Solution

Finally, consider the system:

\[ \begin{aligned} y &= 2x+1\\ y &= -x+4\\ y &= 3x-2 \end{aligned} \]

The three equations are all different, so each contributes new information. Their coefficient matrix has rank 2, which is the maximum possible for a system with two unknowns. However, the augmented matrix has rank 3, indicating that the equations are inconsistent. Consequently, there is no single point where all three lines intersect simultaneously. Although each pair of lines intersects, there is no point that satisfies all three equations, meaning it has no solution.

Another common example of a system with no solution is two parallel lines. Since parallel lines have the same slope but different intercepts, they never intersect. Their equations therefore contradict one another, so their system is inconsistent and has no solution.

Eigenvalues and Eigenvectors

So far, we have seen that matrices can be added, multiplied, inverted, and used to solve systems of equations. Another important question is: does a matrix have any special directions?

Imagine applying a matrix transformation to a vector. Most vectors will change both their length and their direction. However, some special vectors behave differently: after the transformation, they point in exactly the same direction. These special vectors are called eigenvectors. An eigenvector is a nonzero vector whose direction remains unchanged when multiplied by a square matrix. The matrix only stretches, shrinks, or reverses the vector. The amount of scaling is called the eigenvalue. In general:

\[ A\mathbf{x}=\lambda\mathbf{x}, \]

where \(\mathbf{x}\) is an eigenvector and \(\lambda\) is its corresponding eigenvalue. If \(\lambda > 1\), the vector is stretched. If \(0 < \lambda < 1\), it is compressed. If \(\lambda\) is negative, the vector reverses direction. The important point is that the direction of the eigenvector remains unchanged.

Finding Eigenvalues and Eigenvectors

Eigenvalues are found by solving the characteristic equation:

\[ \det(A-\lambda I)=0, \]

where \(I\) is the identity matrix. Solving this equation gives the possible eigenvalues of the matrix. Once an eigenvalue is known, the corresponding eigenvector can be found by solving

\[ (A-\lambda I)\mathbf{x}=0. \]

Each eigenvalue may have one or more associated eigenvectors. For small matrices, these calculations can often be performed by hand. For larger matrices, however, solving the characteristic equation becomes impractical. Instead, numerical algorithms are used to estimate eigenvalues and eigenvectors efficiently.

One of the simplest is the power method, which is an iterative algorithm that repeatedly multiplies a vector by the matrix until it converges to the eigenvector associated with the largest (dominant) eigenvalue. Modern software such as R, Python (NumPy/SciPy), and MATLAB uses more sophisticated numerical algorithms to compute eigenvalues accurately and efficiently.

Eigenvalue Decomposition

Not every square matrix has enough linearly independent eigenvectors. When it does, the matrix is said to be diagonalizable and it can be written in a special form called eigenvalue decomposition:

\[ A=Q\Lambda Q^{-1}, \]

where:

• \(Q\) contains the eigenvectors of \(A\) as its columns.
• \(\Lambda\) is a diagonal matrix containing the corresponding eigenvalues.
• \(Q^{-1}\) inverses the transformation created by \(Q\); transforming the coordinates back to the original basis.

Eigenvalue decomposition separates a complicated matrix transformation into three simpler steps: first transforming into the coordinate system defined by the eigenvectors, then independently scaling each of those directions according to its eigenvalue, and finally transforming back to the original coordinate system.

Singular Value Decomposition (SVD)

Eigenvalue decomposition is extremely useful, but it has an important limitation: it only applies to certain square matrices. A more general decomposition called the Singular Value Decomposition (SVD) works for every matrix, including rectangular matrices.

\[ A=U\Sigma V^{T}, \]

Unlike eigenvalue decomposition, the SVD is constructed from two related eigenvalue problems. The columns of \(V\) are the eigenvectors of \(A^{T}A\), while the columns of \(U\) are the eigenvectors of \(AA^{T}\). The diagonal entries of \(\Sigma\), called singular values, are the square roots of the eigenvalues of these matrices. In this way, SVD extends the ideas of eigenvalues and eigenvectors to every matrix.

The three matrices have distinct roles:

• \(V\) contains the right singular vectors, which describe the most important directions in the input space.
• \(\Sigma\) is a diagonal matrix containing the singular values, which measure how much each of these directions is stretched or compressed.
• \(U\) contains the left singular vectors, which describe the corresponding directions in the output space after the transformation.

Geometrically, SVD decomposes every matrix transformation into three simple operations:

1. Rotate (or change coordinates) into the principal input directions using \(V^{T}\).
2. Stretch or compress independently along those directions using \(\Sigma\).
3. Rotate into the final output coordinate system using \(U\).

Large singular values correspond to directions that preserve most of the information in the data, while very small singular values often represent noise or redundant information. This property makes SVD particularly useful for dimensionality reduction, data compression, and identifying the most important patterns in large datasets.

Because SVD works for every matrix and is numerically stable, it is one of the most important tools in modern statistics, machine learning, and data science. It forms the computational foundation of techniques such as Principal Component Analysis (PCA), recommendation systems, image compression, natural language processing, and many optimization algorithms.

Conclusion

Linear algebra is an incredibly broad field, with entire textbooks devoted exclusively to its theory and applications. This Part I of the series was not intended to cover every aspect of the subject, but rather to provide a solid foundation by introducing some of its most important concepts, ideas, and terminology. My goal has been to equip the reader with the essential language and intuition needed to navigate the subject with confidence.

Now that we have established the fundamental theory of linear algebra, in Part II of this series we will move beyond definitions and explore its practical applications. We will see how the concepts introduced form the backbone of many modern technologies, enabling computers to recognise patterns, make predictions, and solve complex real-world problems efficiently.