Math for AI 101 Warm-up

Acknowledgement: This post is based mainly on Dr. Quang-Du Nguyen’s presentation and slides and partly on on TA, STA, and peers’ responses from the course AIO2025 offered by AI VIETNAM.

Relevance: The audience of this lecture were those who have read Python for AI 106, had had some coding experience, and known a bit of AI knowledge. Absolute beginners are encouraged to read the material and repeatedly practice coding without AI assistance to strengthen their coding and code-reading skills. Note that only necessary Python knowledge is introduced here.

In today’s post, we’ll explore NumPy arrays.

To create and use an array in NumPy, we must include a numpy package with the following syntax.

import numpy as np

np is an alias for users to avoid typing numpy over and over again.

NumPy Array

an_array = np.array([element_1, ..., element_n])

Like a list, an array is a container of multiple elements but it is homogeneous, meaning

• all its elements are of the same data type.
• all its elements are of the same size.
• all its elements are stored continuously.

The data type of all the elements automatically converts to some data type. For example, numpy.str_ if one of them is a string or numpy.float64 if one is a float.

a_list = ['aio', 3.14, True, -1]
an_array = np.array(a_list)

for item in an_array:
    print(type(item))

'''
<class 'numpy.str_'>
<class 'numpy.str_'>
<class 'numpy.str_'>
<class 'numpy.str_'>
'''

A ValueError will be raised if we try to create an array of an integer and a two-element list since their sizes are different.

An array can represent complex and higher dimensional data, for example grayscale or color images. When arrays are used in AI, we tend to think of an array as a generalization of vectors which are displayed horizontally and where there can be more than one set of indices. You can imagine an arrow in space. It is identified by its length and direction which is where the arrow points.

You may wonder, We know some similarities and differences between lists and arrays, but what is it better about arrays then lists?

We’ll discuss the details bits by bits, but for now let’s look at main bullet points.

Indexing, Slicing, and Updating an Element

Indexing, slicing, and updating an element in a NumPy array works similarly as in list. Note that NumPy arrays are displayed without commas in the output.

a_data = np.array([2, 3, 4, 5, 6, 7])

# indexing
a_data[0]
>>> 2

# multi-indexing
a_data[[1, 4, 2]]
>>> [3 6 4]

# slicing 1st method
a_data[1:7:2]
>>> [3 5 7]

# updating an element
a_data[0] = 10
a_data
>>> [10  3  4  5  6  7]

Indexing in arrays is more flexible that lists since we can retrieve as many elements at any index as we want. This multi-indexing technique is called integer array indexing. Thus, indexing for multi-dimensional arrays is less complicated than lists.

a_2d_data = np.array([[1, 2], [3, 4], [5, 6]])

# test for element at position (2, 1)
a_2d_data[2, 1]
>>> 6

Unlike slicing in lists, updating values of a slice of an array changes the original array and vice versa. Imagine a wall part of which you want to place a window on. You start taking bricks out that part so you can put the window in later. The wall is your original array and the soon-to-be-window area is the slice of that array.

a_data = np.array([2, 3, 4, 5, 6, 7])
sub_data = a_data[1:4]
sub_data[0] = 1000

# test
print(f'sub_data: {sub_data}')
print(f'a_data: {a_data}')

'''
sub_data: [1000    4    5]
a_data: [2 1000    4    5    6    7]
'''

arr = np.ones((3,3))
sub_arr = arr[1:, 1:]
print('original:', sub_arr, arr)

# reassign a new array to sub_arr
sub_arr = sub_arr + 1
print('reassign:', sub_arr, arr)

'''
original: [[1. 1.]  [[1. 1. 1.]
           [1. 1.]]  [1. 1. 1.]
                     [1. 1. 1.]]
                   
reassign: [[2. 2.]  [[1. 1. 1.]
           [2. 2.]]  [1. 1. 1.]
                     [1. 1. 1.]]
'''

arr = np.ones((3,3))
sub_arr = arr[1:, 1:]
print('original:', sub_arr, arr)

# updating values of sub_arr
sub_arr += 1
print('update:', sub_arr, arr)

'''
original: [[1. 1.]  [[1. 1. 1.]
           [1. 1.]]  [1. 1. 1.]
                     [1. 1. 1.]]
                    
update: [[2. 2.]  [[1. 1. 1.]
         [2. 2.]]  [1. 2. 2.]
                   [1. 2. 2.]] 
'''

$+$, $*$, and Other Operators

Unlike lists, $+$ and $*$ operators applied to arrays are literal element-wise addition and multiplication. There’s no need to explicitly iterate through arrays. This technique is called vectorization, making complicated calculations significantly efficient and faster than traditional loops. Almost all NumPy operations are inherently vectorized.

data1 = np.array([1, 3])
data2 = np.array([5, 7])

# + operator - vectorization
data1 + data2
>>> [6 10]

# * operator
data_1 * 3
>>> [3 9]

# * operator - vectorization
data1 * data2
>>> [5 21]

Arithmetic, comparison, and logical & and | operators can be flexibly used inside index brackets []. Comparison operators compare elements of an array to a value. It returns an array of boolean values. Using those booleans, we can conditionally select elements and create a new array of values from the original. This technique is called boolean array indexing. It’s imperative that we use parentheses () with logical operators.

a_data = np.array([1, 2, 3, 4, 5, 6, 7, 8, 10])

# > operator
a_data > 7
>>> [False False False False False False False  True  True]

# % and == operators
even_numbers = a_data[a_data % 2 == 0]
even_numbers
>>> [2  4  6  8 10]

# +, <=, >=, and & operators
filtered_elements = a_data[(a_data + 2 >= 5) & (a_data < 9)]
filtered_elements
>>> [3 4 5 6 7 8]

Common Methods

We’ll see that many NumPy methods have no effect on the original array. This is because we use np as an object and pass array_name to it. The object changes but the array doesn’t.

Method/Function	Purpose	Returned Value	Updated
np.ndim(array_name) array_name.ndim	Display the dimension of an array	Number
np.shape(array_name) array_name.shape	Display the shape/size of an array	Tuple of numbers
np.arange(start, end, step)	Create an array within a range	Array	N/A
np.linspace(start, end, number of elements)	Create an array within a range given the number of elements	Array	N/A
np.append(array_name, inserted_value)	Insert an element to the end	New array
np.append(array_name, inserted_list)	Insert more than one element to the end	New array
np.insert(array_name, desired_index, inserted_value)	Insert an element at a desired index	New array
np.append(array_1, array_2)	Join two lists	New array
np.concatenate((array_1, …, array_n))	Join $n$ lists	New array
np.delete(array_name, desired_index)	Delete an element at a desired index	New array
np.delete(array_name, np.arange(start, end, step))	Delete elements at desired indices	New array
np.where(array_name == desired_value)[0]	Display indices of a desired_value	Array of indices
np.sort(array_name)	Sort a given array in an ascending order	New array
array_name.sort()	Sort a given array in an ascending order	None
np.sum(array_name == element_to_count)	Count the number of elements of a given value	Number
array_name.copy() np.copy(array_name)	Make a copy of an array	New array
len(array_name)	Display the length of an array	Number
np.min(array_name)	Display the minimum value	Value
np.max(array_name)	Display the maximum value	Value
np.sum(array_name)	Display the sum of all elements	Number
np.ndenumerate(array_name)	Attach an index and its value	ndenumerate object

Do not confuse an array’s dimension with its shape.

a = np.array([[1, 2, 7], [3, 4, 8], [5, 6, 9]])

# 2D nested array 
a.ndim
>>> 2

# 3-by-3 matrix
a.shape
>>> (3, 3)

We can also use np.append() or np.concatenate() to join parts of two lists.

data1 = np.array([1, 3])
data2 = np.array([5, 7])

# test
np.append(data1[:1], data2[1:])
>>> [1, 7]

np.concatenate((data1[:1], data2[1:]))
>>> [1, 7]

With lists, we must write our own function if we want to delete all elements of the same value. But with arrays, we can utilize built-in methods to make it happen. We know np.where() returns an array of indices of some desired value while np.delete() can delete an element at a desired index. Let’s see how we can combine them to achieve what we wish.

an_array = np.array([2, 7, 1, 6, 5, 9, 3, 5])

# get indices of a desired value
del_indices = np.where(an_array == 5)[0]
del_indices
>>> [4 7]

# test
np.delete(an_array, del_indices)
>>> [2 7 1 6 9 3]

an_array
>>> [2 7 1 6 5 9 3 5]

Notice from above that the original array is not altered after deletion.

a_data = np.array([1, 2, 3, 4, 5, 6, 7, 8, 10])

# test
filtered_indices = np.where((a_data + 2 >= 5) & (a_data < 9))
filtered_indices[0]
>>> [2, 3, 4, 5, 6, 7]

Note that here the returns are indices, not elements themselves as in boolean array indexing above.

Advanced Techniques in NumPy

The integer array indexing, boolean array indexing, and vectorization previously are considered advanced. We’ll discuss more advanced techniques in this section.

Array Reshaping

With arrays, we can change the shape of an array without changing its data. Reshaping carries out the action from left to right and from top to bottom by default. We can specify which index order to reshape an array using order=index_order_type. Feel free to check it out from outside resources.

For the reshaping purpose, we can use either reshape() or flatten(). While flatten() converts an array into a 1D array automatically without any argument, reshape() takes a tuple of numbers as the desired shape. Those numbers must mathematically be factors of the number of elements in the array. Both methods we’ll look at in a moment return new arrays and have no effect on the original.

a_2d_data = np.array([[1, 2], [3, 4], [5, 6]])

# reshape to a (6, 1) matrix
a_2d_data.reshape(6, -1)
>>> [1 2 3 4 5 6]

# reshape to a (2, 3) matrix
a_2d_data.reshape(-1, 3)
>>> [[1 2 3]
     [4 5 6]]

# flatten a matrix to 1D
a_2d_data.flatten()
>>> [1 2 3 4 5 6]

# original matrix
a_2d_data
>>> [[1 2]
     [3 4]
     [5 6]]

Broadcasting: The Unsung Hero

What makes broadcasting so powerful is its ability to extend a smaller array to match the size of a larger one. For example, a 1-by-2 matrix can be extended to a 3-by-2 matrix or a single-element matrix—a number—can be extended to any matrix. Thanks to the feature, we can perform element-wise operations—even when they originally have different shapes.

a = np.array([[1, 2, 7], [3, 4, 8], [5, 6, 9]])
b = np.array([1, 2, 3])

# test for addition
a + b 
>>> [[2  4 10]
     [4  6 11]
     [6  8 12]]

# test for multiplication
a * b
>>> [[1  4 21]
     [3  8 24]
     [5 12 27]]

Stack Two Arrays

data1 = np.array([[1, 3], [2, 4]])
data2 = np.array([[5, 7], [6, 8]])

# vertical result
[[1 3]
 [2 4]
 [5 7]
 [6 8]]

# horizontal result
[[1 3 5 7]
 [2 4 6 8]]

There is another way to stack two arrays called depth-wise stacking using np.dstack(). We won’t talk about it here but you can check it out.

Compare Element-Wise Arrays

In the example we’re about to go over, we aim to get an array of greater elements at corresponding positions from two given arrays.

data1 = np.arange(10).reshape(2, -1)
data2 = np.array([2, 7, 1, 6, 5, 9, 3, 5, 8, 0]).reshape(2, -1)

# output
[[2 7 2 6 5]
 [9 6 7 8 9]]

There are three methods to go about this problem.

np.maximum(arr_1, arr_2)

np.maximum(data1, data2)

np.where(if_condition, if_result, else_result)

np.where(data1 > data2, data1, data2)

The logical operators in if_condition are element-wise performed. This condition returns a mask boolean.

np.vectorize()

# compare elements of two arrays
def get_max(x, y):
    if x >= y:
        return x
    else: 
        return y

# vectorize function
element_wise_compare = np.vectorize(get_max, otypes=[int])

# test
element_wise_compare(data1, data2)

np.vectorize() replaces a for loop, looking more organized.

Mathematical Matrix Operations

Norm/Length of a Vector is calculated by the formula \(\left \lVert \vec{u} \right \rVert = \sqrt{u \cdot u} \text{ where } \vec{u} = \begin{bmatrix} u_1 \\ \vdots \\ u_n \end{bmatrix}\).

u = np.array([1, 2, 4, 2])

np.linalg.norm(u)
>>> 5.0

Vector Addition can be done by either np.add() or $+$ operator discussed above. \(\vec{u} + \vec{v} = \begin{bmatrix} u_1 + v_1 \\ \vdots \\ u_n + v_n \end{bmatrix} \text{ where } \vec{u} = \begin{bmatrix} u_1 \\ \vdots \\ u_n \end{bmatrix} \text{ and } \vec{v} = \begin{bmatrix} v_1 \\ \vdots \\ v_n \end{bmatrix}\).

u = np.arange(5)
v = np.repeat(3, 5)

# test
np.add(u, v)
>>> [3, 4, 5, 6, 7]

Hadamard Product produces element-wise multiplication just like what $*$ operator does. \(\vec{u} \odot \vec{v} = \begin{bmatrix} u_1 \times v_1 \\ \vdots \\ u_n \times v_n \end{bmatrix} \text{ where } \vec{u} = \begin{bmatrix} u_1 \\ \vdots \\ u_n \end{bmatrix} \text{ and } \vec{v} = \begin{bmatrix} v_1 \\ \vdots \\ v_n \end{bmatrix}\). This product is commutative, meaning $\vec{u} \odot \vec{v} = \vec{v} \odot \vec{u}$.

u = np.arange(5)
v = np.repeat(3, 5)

# test
np.multiply(u, v)
>>> [0,  3,  6,  9, 12]

Dot Product
Let’s first look at two ways to compute this product for vectors which multiplies element-wise arrays and sum the resulted elements up. \(\vec{u} \cdot \vec{v} = u_1 \times v_1 + \cdots + \ u_n \times v_n \text{ where } \vec{u} = \begin{bmatrix} u_1 \\ \vdots \\ u_n \end{bmatrix} \text{ and } \vec{v} = \begin{bmatrix} v_1 \\ \vdots \\ v_n \end{bmatrix}\).

u = np.arange(5)
v = np.repeat(3, 5)

# test for dot()
u.dot(v)
>>> 30

# test for np.dot()
np.dot(u, v)
>>> 30

We can also compute this product for matrices in two methods one of which is np.dot(). The formula will be discussed in Math for AI 101.

A = np.array([[1, 3], [2, 4]])
B = np.array([[5, 7], [6, 8]])

# test for np.dot()
np.dot(A, B)
>>> [[23 31]
     [34 46]]
     
# test for @ operator
A @ B
>>> [[23 31]
     [34 46]]

Practice Exercises

Exercise 1: Write code to result in the following pictures.

Exercise 2: Write code to stack two arrays vertically and horizontally for each method in the example above.
Exercise 3: Write code to compare the two arrays above and output smaller elements in three ways.

FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.

Questions	Answers
What are NumPy and tensor for?	They are used to store data like images and texts and then pass to models to train.
How do I sort for more than one dimension array?	You need to specify the axis.
Methods np. don’t change the original array. Why is it important?	That the original array is untouched prevents memory congestion and preserves speed.
What is mask boolean used in?	It is common in ML/DL, for example image processing.
Rules for broadcasting	Rule 1: Add leading 1s to match dimensions -> If two arrays have a different number of dimensions, NumPy will pad the smaller array with 1s on the left (leading dimensions) so both arrays have the same number of dimensions. -> (3, 4) and (4,) → (3, 4) and (1, 4) Rule 2: Dimensions must match or be 1 -> For each dimension, the sizes must be equal or one of them must be 1 (in which case the array is broadcast along that dimension) -> (3, 1) and (1, 4) → both broadcast to (3, 4) Rule 3: Incompatible shapes cause an error -> If in any dimension, the sizes are not equal and neither is 1, broadcasting fails and NumPy raises an error -> (3, 2) and (3, 4) → Error (2 ≠ 4 and neither is 1).