Math for AI 102

Acknowledgement: This post is based mainly on Dr. Quang-Vinh Dinh’s presentation and slides and partly on on TA, STA, and peers’ responses from the course AIO2025 offered by AI VIETNAM.

Relevance: The audience of this lecture were those who have read Math for AI 101, had some coding experience, and known a bit of AI knowledge. Absolute beginners are encouraged to read the material and repeatedly practice coding without AI assistance to strengthen their coding and code-reading skills. Note that only necessary Python knowledge is introduced here. Also, not every single detail from Linear Algebra is introduced in this Math for AI series. Instead, we’ll learn pick up more material along the way.

The simplest ideas reveal what is most essential and enduring.
-Dr. Quang-Vinh Dinh

In today’s post, we’ll explore matrices and cosine similarity. We’ll also discuss some applications in AI including background substraction, traffic sign matching, and K-nearest neighbors.

All data can be transformed into vectors, so Linear Algebra can handle most of the real-world information.

Vector and Matrix

A vector is a collection of real numbers and has a length of n, that is, \(\mathbf{u} = \vec{u} = \begin{bmatrix} u_1 \\ \vdots \\ u_n \end{bmatrix} \in \mathbb{R}^n\).

A matrix is a collection/generalization of vectors and has m rows and n columns, that is, \(A = \begin{bmatrix} a_{11} &\cdots& a_{1n} \\ \vdots &\ddots& \vdots \\ a_{m1} &\cdots& a_{mn} \end{bmatrix} \in \mathbb{R}^{m \times n}\).

Mathematically, a linear equation is an equation where the variables/unknowns’ power is at most one. Its graph for one variable is called a line, for two variables is a plane, and for more than two is a hyperplane. For example, if $x_1, \cdots, x_n$ are variables, then some linear equations are

System of Linear Equations

Assume that a company’s sales depends on their advertising expenses on TV, radio, and newspaper and specifically computed based on the linear equation formula

where $a_1, a_2, a_3, \text{ and } b$ are fixed for specific sets of T, R, and N. If we know all these parameters and expenses, we can easily figure out S.

For the sake of simplicity, we can exclude b or set b to be 0. If there are three sets of T, R, N, and S, we can construct a system of linear equations

We can also rewrite this system using vectors

and a matrix

For this matrix-vector calculation, each linear equation or each entry in the resulting vector is obtained by element-wise multiplying the corresponding row of the matrix with the vector and summing the results. Each multiplication is known as dot product, which will be introduced later.

Generally, we can have more unknowns $a_i’s$ than equations or vice versa and, regardless of circumstance, write

where $A$ is a data matrix, $\mathbf{x}$ is an unknown, $\mathbf{b}$ is a data vector, and the number of columns in $A$ must match that of the unknown vector $\mathbf{x}$.

If we have more unknowns than equations, then there are infinitely many solutions. In such cases, our goal is to figure out the most optimized solution.

Vectors Operations

For scalar $\alpha$, \(\mathbf{u} = \begin{bmatrix} u_1 \\ \vdots \\ u_n \end{bmatrix} \in \mathbb{R}^{n}\), and \(\mathbf{v} = \begin{bmatrix} v_1 \\ \vdots \\ v_n \end{bmatrix} \in \mathbb{R}^{n}\), we have the following vector operations.

Operation	Formula	Commutative
Vector Addition element-wise addition $+$ np.add(vec1, vec2) np.add(vec2, vec1)	$$\mathbf{u} + \mathbf{v} = \begin{bmatrix} u_1 + v_1 \\ \vdots \\ u_n + v_n \end{bmatrix}$$	$$\mathbf{u} + \mathbf{v} = \mathbf{v} + \mathbf{u}$$
Vector Subtraction element-wise subtraction $-$ np.subtract(vec1, vec2)	$$\mathbf{u} - \mathbf{v} = \begin{bmatrix} u_1 - v_1 \\ \vdots \\ u_n - v_n \end{bmatrix}$$
Hadamard Product element-wise multiplication $*$ np.multiply(vec1, vec2) np.multiply(vec2, vec1)	$$\mathbf{u} \odot \mathbf{v} = \begin{bmatrix} u_1 v_1 \\ \vdots \\ u_n v_n \end{bmatrix}$$	$$\mathbf{u} \odot \mathbf{v} = \mathbf{v} \odot \mathbf{u}$$
Hadamard Division element-wise division $/$ np.divide(vec1, vec2)	$$\mathbf{u} \oslash \mathbf{v} = \begin{bmatrix} u_1 / v_1 \\ \vdots \\ u_n / v_n \end{bmatrix}$$
Scalar Multiplication $*$	$$\alpha \mathbf{u} = \alpha \begin{bmatrix} u_1 \\ \vdots \\ u_n \end{bmatrix} = \begin{bmatrix} \alpha u_1 \\ \vdots \\ \alpha u_n \end{bmatrix}$$	$$\alpha \mathbf{u} = \mathbf{u} \alpha$$
Transpose convert rows into columns vec.T	$$\mathbf{u}^\intercal = [u_1 \cdots u_n] \in \mathbb{R}^{1 \times n}$$
Dot Product np.dot(vec1, vec2) np.dot(vec2, vec1) vec1.subtract(vec2) vec2.subtract(vec1)	$$\begin{align*} &\mathbf{u} \cdot \mathbf{v} = \langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^\intercal \mathbf{v} \\ &= u_1 \times v_1 + \cdots + \ u_n \times v_n \end{align*}$$	$$\mathbf{u} \cdot \mathbf{v} = \mathbf{v} \cdot \mathbf{u}$$
Norm/Length of a Vector np.linalg.norm(vec) normally use squared norm $\left \lVert \mathbf{u} \right \rVert ^2$	$$\begin{align*} &\left \lVert \mathbf{u} \right \rVert = \sqrt{\mathbf{u} \cdot \mathbf{u}} \\ &= \sqrt{u_1^2 + \cdots u_n^2} \end{align*}$$

Geometrically, dot product of two vectors is based on the projection of the first vector onto the second, that is, represent how much the former points in the same direction as the latter. The more positive and higher the result is, the closer they are, the more similar directions they point towards and vice versa. If the result is 0, then two vectors are perpendiclar/orthogonal.

Matrix Operations

For \(A = \begin{bmatrix} a_{11} &\cdots& a_{1n} \\ \vdots &\ddots& \vdots \\ a_{m1} &\cdots& a_{mn} \end{bmatrix} \in \mathbb{R}^{m \times n}\), \(B = \begin{bmatrix} b_{11} &\cdots& b_{1q} \\ \vdots &\ddots& \vdots \\ b_{p1} &\cdots& b_{pq} \end{bmatrix} \in \mathbb{R}^{p \times q}\), and \(\mathbf{x} = \begin{bmatrix} x_1 \\ \vdots \\ x_n \end{bmatrix} \in \mathbb{R}^{n}\), we have the following operations.

Operation	Formula	Commutative
Matrix Addition element-wise addition	For $m = p$ and $n = q$, $$A + B = \begin{bmatrix} a_{11} + b_{11} &\cdots& a_{1n} + b_{1q} \\ \vdots &\ddots& \vdots \\ a_{m1} + b_{p1} &\cdots& a_{mn} + b_{pq} \end{bmatrix}$$	$$A + B = B + A$$
Vector Subtraction element-wise subtraction	For $m = p$ and $n = q$, $$A - B = \begin{bmatrix} a_{11} - b_{11} &\cdots& a_{1n} - b_{1q} \\ \vdots &\ddots& \vdots \\ a_{m1} - b_{p1} &\cdots& a_{mn} - b_{pq} \end{bmatrix}$$
Transpose convert rows into columns mat.T	$$A^\intercal = \begin{bmatrix} a_{11} &\cdots& a_{m1} \\ \vdots &\ddots& \vdots \\ a_{1n} &\cdots& a_{mn} \end{bmatrix} \in \mathbb{R}^{n \times m}$$
Matrix-Vector Multiplication transform a vector using a matrix np.dot(mat, vec) mat @ vec mat.dot(vec)	$$\mathbf{c} = A \mathbf{x} = \begin{bmatrix} a_{11} x_1 + \cdots + a_{1n} x_n \\ \vdots \\ a_{m1} x_1 + \cdots + a_{mn} x_n \end{bmatrix}$$ $$c \in \mathbb{R}^m \text{ where } c_i = \sum_{j=1}^{n} a_{ij} x_j$$
Matrix-Matrix Multiplication transformations B -> A np.dot(mat1, mat2) mat1 @ mat2 mat1.dot(mat2)	For $n = p$, $$\begin{align*} &C = AB \\ &= \begin{bmatrix} \sum_{j=1}^n a_{1j} b_{j1} &\cdots& \sum_{j=1}^n a_{1j} b_{jq} \\ \vdots &\ddots& \vdots \\ \sum_{j=1}^n a_{mj} b_{j1} &\cdots& \sum_{j=1}^n a_{mj} b_{jq} \end{bmatrix} \end{align*}$$ $$C \in \mathbb{R}^{m \times q} \text{ where } c_{ik} = \sum_{j=1}^{n} a_{ij} b_{jk}$$

Notation and Other Operations
$X \in \mathbb{R}^{m \times n}$ is also denoted by $X \sim (m, n)$. This is also applied to higher-dimensional matrices.

We can also take exponential function or square root of a matrix as follows.

x = np.arange(6).reshape((2,3))
print(f'x is {x}')
print(f'square root of x is {np.sqrt(x)}')
print(f'exponential of x is {np.exp(x)}')

'''
x is [[0 1 2]
      [3 4 5]]
square root of x is [[0.         1.         1.41421356]
                     [1.73205081 2.         2.23606798]]
exponential of x is [[  1.           2.71828183   7.3890561 ]
                     [ 20.08553692  54.59815003 148.4131591 ]]
'''

argmin and argmax
Let $\mathbf{x}$ be and array and $D$ be its elements’ indices.

Summation and Average
We use np.sum() and np.mean() to compute the summation and average of values of a vector. The mathematical formulae for these methods are as follows.

Applications in AI

We know from the previous post that

• a color image is a 3D matrix of values between 0 and 255, (Height, Width, channel) where there are 3 color channels RGB
• a grayscale image is a 2D matrix of values between 0 and 255, (Height, Width).

Rotate Left an Image (Perpendicular)
We’ve learned how to handle a 2D matrix above, so rotating left a grayscale image should be feasible just using some above operations.

About the case of a color image—a 3D matrix, we know the last dimension is always 3 since it contains color channels. But we do want to swap its height and width. We can certainly do this using the idea of array indexing. The hint is that we can use the NumPy method np.transpose().

Convert Color to Grayscale Image

In fact, OpenCV provides a function that can convert a color to grayscale image. The example above demonstrates behind-the-scene computation.

color_image = cv2.cvtColor(color_image, cv2.COLOR_BGR2GRAY)

Background Subtraction
In OpenCV, there is a function called cv2.threshold(), which works quite similarly to np.clip() or np.where() in NumPy. This function is specifically used for image data, not ndarray in general. Be mindful about the proper data type that it can handle.

import cv2

_, array_output = cv2.threshold(array_input, threshold, max_value, cv2.THRESH_BINARY)

This statement returns two outputs including the threshold and the thresholded image known as mask, respectively. Note that the threshold and max_value are numbers but need not be integers. According to THRESH_BINARY, we can interpret the condition statement that

Cosine Similarity

A loss function is a part of the model training process and cosine similarity can be used for such a function.

Dot Product

In addition to length/norm, a vector is always associated with only one direction. Two vectors can have similar or different directions. For example, scale of a vector have different length but the same direction as itself. Practically, we can think words like teachers, students, and banks. The first two may be arranged in relevant categories, so their vectors may point in closer directions. On the other hand, the last two may not.

As mentioned previously, the dot product of two vectors is geometrically based on the projection of one vector onto another. This contributes to the geometric definition in the table below. The interactive figure above illustrates

• the smaller the angle between two vectors is
• the closer they are
• the longer the projection of one vector onto the other is
• the more they points in the same direction.

If the result is negative, their directions are different or opposite. If it is positive, they are similar or the same. If it is 0, they are perpendicular. Therefore, dot product actually represents how much one vector points in the same direction as the other. This refers to the similarity of two vectors, which goes together with cosine similarity. In fact, the definition of dot product between two vectors involves the cosine of their smaller angle.

Cosine Similarity

Cosine similarity is used to measure the similarity between two vectors. From two definitions above, we have the formula for cosine of the smaller angle between two vectors

Three important property for $\cos(\mathbf{u}, \mathbf{v})$ are

1. The more similar two vectors are, the closer to 1 $\cos(\alpha)$ is.
2. $\cos(\mathbf{u}, \mathbf{v}) = \cos(a \mathbf{u}, b \mathbf{v}) \text{ for } ab > 0$.
3. $\cos(\mathbf{u}, \mathbf{v}) \neq \cos(\mathbf{u} + c, \mathbf{v} + d)$.

Above we have an example of words that shows usefulness of cosine similarity. Moreover, this concept can be applied to spot similar images.

• We know that we can use basic operations to change brightness of an image.
• Obviously, images with different brightness are still the same image.
• If we use absolute difference on them, a computer will distinguish them as dissimilar.
• Using cosine similarity can address this issue thanks to the second property.

Applications in AI

A unit vector is a vector of length 1. The unit vector in the direction of $\mathbf{x}$ is the vector $\frac{\mathbf{x}}{\left \lVert \mathbf{x} \right \rVert}$.

The distance between two points $x, y \in \mathbb{R}^n$ is the length of the vector from $\mathbf{x}$ to $\mathbf{y}$.

While the absolute difference of two vectors is a vector, the distance between their heads (two points) is the length/norm of the resulting vector.

Traffic Sign Matching

One real-world application for absolute difference, distance, and cosine similarity is checking if two traffic signs are similar or not. They are similar when

1. Absolute difference and distance are small and close to 0
   • Issue: The flip of an image is concluded to be more similar than its brighter image. This is because the results for the former may be smaller than that for the latter.
2. Cosine similarity is close to 1.
   • Address the Issue: The brightness of an image has no significant effect on images’ similarity.

Figures below demonstrates the problem with absolute difference and distance and a solution using cosine similarity.

What if two images are the same but one is a rotation of the other?

K-Nearest Neighbors (KNN)

KNN is a popular algorithm in Machine Learning. It involves considering each piece of data as a vector and computing the distance between a testing point and training points. As such, the algorithm can select the K nearest neighbors and make further analysis to provide an suitable ouput.

FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.