Relevance: The audience of this lecture were those who have read Machine Learning 101 Warm-up, had some coding experience, and known a bit of AI knowledge. 2025-08-05-KNN-pre101

In today’s post, we’ll explore KNN and Supervised Learning. We’ll also discuss some applications in AI including a text classification problem and a medical case study.

Machine Learning

Machine Learning (ML) is a subfield of Artificial Intelligence (AI), referring to learning the best approximation of a function from the training data so that data can be predicted accurately.

mindmap
    root)Machine Learning(
        ((<strong>Semi-Supervised<br/> Learning</strong>))
        ((<strong>Supervised<br/> Learning</strong>))
            </strong>Classification</strong>
                (Identity Fraud Detection)
                (Image Classification)
                (Customer Retention)
                (Diagnostics)
            <strong>Regression</strong>
                (Advertising Popularity Prediction)
                (Weather Forecasting)
                (Markekt Forecasting)
                (Estimating Life Expectancy)
                (Population Growth Prediction)
        ((<strong>Unsupervised<br/> Learning</strong>))
            <strong>Clustering</strong>
                (Recommender Systems)
                (Targetted Marketing)
                (Customer Segmentation)
            <strong>Dimensionality<br/> Reduction</strong>
                (Big Data Visualization)
                (Meaningful Compression)
                (Structure Discovery)
                (Feature Elicitation)
        ((<strong>Reinforcement<br/> Learning</strong>))
            (Real-time Decisions)
            (Robot Navigation)
            (Learning Tasks)
            (Skill Acquisition)
            (Game AI)

There are four main types of ML:

• Supervised Learning (SL): learning with guidance—from a labeled dataset to map features to a potentially correct label.
  • Examples: predicting the shape of a geometric object after learning a labeled dataset and predicting the linear relationship between features and label.
• Unsupervised Learning (USL): learning without explicit guidance or labeled data to stamp labels for data points.
  • Examples: labeling a dataset of animal images.
• Semi-Supervised Learning (SSL): a combination of SL and USL.
• Reinforcement Learning (RL): learning via interactions with external environments and evaluation-reward-punishment process to maximize the cumulative reward. This is a trial-and-error approach where agents, just like a human, learn from feedback and adjust its actions until getting a desired outcome.
  • Examples: autonomous vehicles or chess playing.

flowchart TB
    subgraph OutputTitle[Output]
        direction LR
        mapping(Mapping):::output
        classes(Classes):::output
        actions(Actions):::output
    end

    subgraph InputTitle[Input]
        direction TB
        labeledData(Data with Labels):::input
        unlabeledData(Data without Labels):::input
        statesActions(States and Actions):::input
    end

    labeledData --> SL ----> mapping
    criticsSL((" ")):::cSL -->|Error| SL(Supervised Learning):::SL -.->|Targets| criticsSL
    unlabeledData --> USL(Unsupervised<br/> Learning):::USL ----> classes
    statesActions --> RL(Reinforcement<br/> Learning):::RL ----> actions
    criticsRL((" ")):::cRL -->|Reward| RL -.->|Evaluations| criticsRL

    classDef input fill:#fec3ec,stroke:#7293ba,stroke-width:2px,font-family:serif;
    classDef output fill:#99ffff,stroke:#628e6e,stroke-width:2px,font-family:serif;
    classDef outputHeader fill:#bff3fd,stroke:#3b6fa6,stroke-width:2.5px,font-weight:bold,text-align:center;
    classDef inputHeader fill:#f8dcfe,stroke:#cd7a39,stroke-width:2.5px,font-weight:bold,text-align:center;
    classDef SL fill:#adff00,stroke:#7293ba,stroke-width:2px,font-family:serif,font-weight:bold;
    classDef USL fill:#fffb96,stroke:#7293ba,stroke-width:2px,font-family:serif,font-weight:bold;
    classDef RL fill:#b967ff,stroke:#7293ba,stroke-width:2px;
    classDef cSL fill:#74d600,stroke:#7293ba,stroke-width:1px;
    classDef cRL fill:#962fbf,stroke:#7293ba,stroke-width:1px;
    classDef blockHeader fill:none,stroke-width:0px;

    class OutputTitle outputHeader
    class InputTitle inputHeader
    class SLblock blockHeader
    class RLblock blockHeader

Approaches

There are several appproaches for a ML problem.

• Parametric Approach: This is a classic SL. We try to parametrize the function, so we need to go through some important steps.
  • Identify if the function is linear or non-linear.
  • Identify the parameters for a linear function/regression.
• Eager Learning Approach: We build and train a model/function based on the entire training dataset before the prediction phase starts.
  • Spend a long time learning and building the function.
  • Spend less time classifying or inferring the function.
• Model-based Learning Approach: We build a model that captures parttern in the data.
  • Spend a long time learning and building the function.
  • Spend less time classifying or inferring the function.

Eager and model-based learning methods can be both parametric and non-parametric.

$K$-Nearest Neighbors (KNN) Overview

The parametric approach technically involves the assumptions of having/looking for a fixed number of parameters regardless of dataset size. This means the method strongly relies on the assumptions about the data. which could lead to poor performance if the assumptions are incorrect. This is where KNN comes in as a non-parametric approach.

Motivation

KNN is an algorithm where

• there is no assumptions made about underlying data distribution.
• learning takes less time and testing takes more time.
  • a training dataset is stored without learning a function or building a model during the training phase.
  • predictions are made on data only when KNN actually receives test data.

These properties are the reasons for KNN to be called lazy learning or instance-based learning. You can think of it as a lazy student partaking in an exam with no studying or revising in advance.

A typical example is a given image dataset of different kinds of insects and animals with known categories. If we are given a covered image with its intensity matrix and need to predict which group it belongs to, KNN will look for its $K$ nearest neighbors from the dataset to choose a potentially correct category. “Nearest” refers to the closet distance.

Procedure

:pushpin: Note: The value of $K$ can completely change your model’s prediction. That’s because KNN picks the category by majority vote among the $K$ closet neighbors. Thus, choosing the optimal $K$ plays a critical role in KNN.

We should always preprocess the data befor running KNN. Once ready, the algorithm follows a simple flow:

1. Initialize the value of $K$.
2. Compute the similar feature/distance between test point and all data points (every row of the training dataset).
3. Sort the calculated distances in ascending order.
4. Select top $K$ elements from the sorted array.
5. Vote the most frequent label of these elements.
6. Predict—return the predicted class.

flowchart LR
    A((<strong>Initialize k</strong>)) ---> B((<strong>Compute distances</strong></br> between test data</br> and training data)) ---> C((<strong>Sort distances</strong></br> in ascending order)) ---> D((<strong>Get top k elements</strong></br> from the sorted array)) ---> E((<strong>Get the most frequent class</strong></br> of these elements)) ---> F((<strong>Return predicted class</strong>))

In step 5, we may run into a situation where two or more labels appear equally often. This occurs especially when $K$ is even. We’ll explore strategies to handle this later.

For step 2, there is no generic formula for choosing a metric. A good practice is to understand the pros and cons of each metric to decide if it is suitable for the dataset. Also, we can experiment them to see which metric gives the most consistent and meaningful results for the dataset. Common metrics to compute the similarity between features are

• cosine similarity
• correlation coefficient
• L1 (absolute value)
• L2 (Euclidean distance)
• Hamming distance
• Manhattan distance
• Minkowski distance
• Chebyshev distance
• Jaccard distance
• Haversine formula
• Sørensen–Dice coefficient.

DEMONSTRATION FOR METRIC from slides maybe

We can use an available package for KNN with its default metric being Minkowski distance.

# call the package with initialized K
# specify which metric via p, p=2 indicated Euclidean distance
knn = KNeighborsClassifier(n_neighbors=K, p=2)

# save and arrange dataset
knn.fit(X_train, y_train)

# make predictions
pred = knn.predict(X_test)

Selecting $K$ in KNN

As mentioned earlier, different $K$ may give different results. Thus, it’s imperative we choose $K$ that provides the best approximated outcome. One approach is by trial and error—figuring out the value of $K$ holding the highest accuracy after the training process.

:question: This boils down to the question, How can we evaluate a model’s performance based on the choice of $K$?

We can use the following ready-to-use package for this purpose.

from sklearn.metrics import accuracy_score, classification_report

# compute the accuracy percentage
accuracy_score(y_test, y_pred)

# compute the set of precision, recall, and f1-score
classification_report(y_test, y_pred)

Evaluating a Model’s Performance

In practice, we use the values of precision, recall, and F1-score to assess a model’s performance. This is because the (naive) accuracy is not a reliable metric to evaluate a model’s performance. A very high accuracy may lead to a behavior called overfitting. See the analysis below.

Precision is the ratio of all the model’s positive predictions that are actually positive. In the example above, it is the fraction between correctly predicted cancer cases and all the predicted cancer cases.

Recall is the proportion of all the actual positive cases that are correctly predicted positive. In this example, it is the percentage of correctly predicted cancer cases over all the real cancer cases.

$F_1$ score is a metric that combines both precision and recall into a single number using the harmonic mean. This mean penalizes the imbalance of a dataset like the simple algorithm example earlier. That is, $F_1$ score will only be high if both precision and recall are high. We use $F_1$ score to evaluate a model’s performance since it delivers a single quantitative measure that balances precision and recall.

The following table illustrates the kind of problem we may encounter if we only make guesses from precision and recall values. We can choose the algorithm with higher $F_1$ score for a better model performance.

Weight in KNN

In the previous section, we mentioned the issue with the classic KNN algorithm where a tie in voting happens at step 5, especially $K$ is even. In this case, there is no specific instruction on which label should be chosen as two of them are both most frequent.

To address this problem, scikit-learn allows us to provide some additional parameter to control how ties are resolved in the case of some labels making an equal contribution among the $K$ nearest neighbors. This is to stress on each label’s amount of say—its influence compared to others.

The addition parameter discussed here is weights in classifier = KNeighborsClassifier(n_neighbors=8, weights = 'WeightType'). Common weight types are

• uniform: default weights.
• distance
  • Closer neighbors have more influence than farther ones.
  • Weight is typically inverse of distance, that is, $\sum \frac{1}{d}$ where d is the distance from the test point to a training point.
• User-defined.

Practice Exercises

Exercise 1: Write a function to compute Euclidean distance between two data points (vectors) using a for loop.
Exercise 2: Do the same as exercise 1 for $x\_test = (2.4, 0.8)$ and $x\_test = (2.6, 0.8)$ when

1. $k = 1$
2. $k = 3$
3. $k = 5$

Exercise 3: Write code for both exercises above.
Exercise 4: Use the following table and step-by-step procedure with Euclidean distance to predict the salary for $experience = 5.3$ when

1. $k = 2$
2. $k = 3$
3. $k = 4$

Exercise 5: Write code for exercise 4 above.

KNN for Classification Tasks

BoW is simpler and more basic approach to text representation than TF-IDF. BoW only considers word occurences while TF-IDF provides more informative representation of text—taking into account both word importance and context within the entire dataset. TF-IDF is generally preferred. We’ll delve deeper into the two techniques right away.

Bag-of-Words (BoW)

Problem statement: We have the following corpus.

Doc	Label
you reap what you sow	1
beggars can't be choosers	1
like bees to honey	1
bite the hands that feed you	0
a taste of your own medicine	0
burn your bridges behind you	0

Idea: BoW is to investigate distances between two sentences based on word occurences.

1. Tokenization splits text into a list of smaller parts or separate words.
2. Initialize an empty vocabulary. Add new words from tokenized corpus into the vocabulary.
Vocabulary =
3. Take a random string: "you can't pick and choose".
4. Convert the string to a numeric value by creating a BoW from the string and assigning a number to each of them. Two common ways are
• Binary BoW: Words in the string are assigned $1$ and others are $0$.
• (Complete) BoW: Each value is the number of its' word occurrency.
We use the (complete) BoW for this problem.
• From here, we build a matrix where each row is a word in the constructed vocabulary, each column is a sentence from the corpus, and each cell of a column is the BoW value of the corresponding sentence. Corpus sentences now become training vectors/points.
• With the random string, we can build a test vector/point.
DEMONSTRATION for matrix

Term Frequency-Inverse Document Frequency (TF-IDF)

Problem statement: We use the same corpus as in BoW. We can also use this approach to do spam email detection.
Breakdown:

• A document is a sentence/ an email.
• The corpus is a collection of documents.

Idea: TF-IDF is to compute the importance level of a word or phrase relative to its parent sentence/email and the entire corpus. Thus, TF-IDF helps identify words that are significant in a specific document but maybe rare across all the documents.

1. We build the same matrix for the entire corpus as in Bag-of-Words technique.
2. We build three other matrices:
• TF matrix: Each cell is the corresponding TF value of the row word and the column document.
• IDF matrix: Each cell is the corresponding IDF value of the row word and the column document.
• TF-IDF matrix: Each cell is the corresponding TF-IDF value of the row word and the column document.
• Normalized TF-IDF matrix: Each cell is the corresponding normalized TF-IDF value of the row word and the column document.
3. We build vectors for test documents.
4. Term Frequency (TF) measures how often a term/word appears in a document, that is how important a word is in a document.
$$\begin{align*} TF_{(t,d)} &= count(t,d) \\ &= \frac{\text{Number of times word$_\textbf{i}$ appears in a document}}{\text{Total number of words in the document}} \end{align*}$$ or in practice, $$TF_{(t,d)} = \log{(count(t,d) + 1)}$$ However, a word with a high occurence may not actually be of large importance, for example "the", "and", "are", and "is". We need additional step to handle this situation.
5. Inverse Document Frequency (IDF) measures how important a word is across the entire corpus.
$$\begin{align*} IDF_{t} &= \log \frac{\text{Total number of documents}}{\text{Number of documents containing word$_\textbf{i}$}} \\ &= \frac{N}{DF_t} \end{align*}$$ or in practice, $$\begin{align*} IDF_{t} &= \log \frac{\text{Total number of documents + 1}}{\text{Number of documents containing word$_\textbf{i}$ + 1}} + 1 \\ &= \frac{N + 1}{DF_t + 1} + 1 \end{align*}$$ Adding $1$ is to avoid $0$ as a result since it could kill of the importance of a word and avoid division by zero. This technique is called smoothing used in packages like scikit-learn.
6. TF-IDF computes the importance of a word in an email relative to the entire collection.
$$TF-IDF_{(t,d)} = TF_{(t,d)} \times IDF_{t}$$ This multiplication helps compensate for the disproportionate influence of frequently occuring but uninformative words and ensure the more meaningful features contribute appropriately to the distance calculations.
This next step is not required. The TF-IDF matrix contains many zeros and possiblly much greater values, so we need to normalize them like we perform scaling in KNN. This helps avoid bias from document length.
7. Normalized TF-IDF takes the unit vector of each TF-IDF vector. L2 norm is also known as Euclidean norm.
$$L2\_norm(TF-IDF) = \frac{TF-IDF}{\left \| TF-IDF \right \|_2} = \frac{\langle x_1, \cdots, x_n}{\left( \sum_i=1^n x_i^2\right)^\frac{1}{2}}$$ DEMONSTRATION for matrix

Case Study: Code Implementation

We primarily use the scikit-learn library for ML problems.

Problem statement: Build a system to predict whether a tumor is maglinant or benign based on real-valued information using KNN.

Set Up

# for a tabular dataset
import pandas as pd

# this dataset is available in the package, so we just need to import it
from sklearn.datasets import load_breast_cancer

# 
from sklearn.model_selection import train_test_split

# for scaling data (mentioned above)
from sklearn.preprocessing import StandardScaler

# KNN algorithm in the package
from sklearn.neighbors import KNeighborsClassifier

# for classification tasks and assessing how accurate our model is
from sklearn.metrics import accuracy_score, classification_report

The pipeline for a ML problem follows the basic procedure below. In practice, there can more steps to work through.

1. Data input
2. Data preprocessing
3. Data spliting:
   • Training data
   • Test data
4. Data training
5. Model evaluation

Data Input

# load and read the dataset
cancer = load_breast_cancer()

# convert the feature columns to a DataFrame
X = pd.DataFrame(cancer.data, columns=cancer.feature_names)

# convert the label column to a Series
y = pd.Series(cancer.target, name='target')

DEMONSTRATION FOR FIRST FIVE ROWS

Since the target contains non-numerical values: benign and malignant, we need to assign numeric values to them: $1$ for benign and $0$ for malignant.

Data Spliting
We mentioned splitting a dataset (label included) into two sets: training and test data. But in practice, a dataset is usally divided into three following sets:

• Training data
• Test data
• Evaluation data: we use this set to evaluate how accurate the ML model is—checking how close its predicted label values is to its actual label values, then check the same for test data.

val_size = 0.2
test_size = 0.125
random_state = 0
is_shuffle = True

# split training and test data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, 
    test_size=test_size, 
    shuffle=is_shuffle,
    random_state=random_state)

# split training and evaluation data
X_train, X_val, y_train, y_val = train_test_split(
    X, y, 
    test_size=val_size, 
    shuffle=is_shuffle,
    random_state=random_state)

Data Training

# scale data
# call scaling function
scaler = StandardScaler()

# scale both training and test data
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# initialize k
k = 5

# call KNN algorithm
knn_classifier = KNeighborsClassifier(n_neighbors=k)

# train data
knn_classifier.fit(X_train_scaled, y_train)

DEMONSTRATION IN SLIDE, ONLY TAKE 2 FOR VISUALIZATION, SHOULD BE 30 BUT THERE IS NO WAY TO VISUALIZE 30 DIMENSIONS

Model Evaluation
This step plays a very important role during the model development. Depending on the result, we can tune parameters, try different metrics, process the data differently or other methods. Once we get high-enough accuracy, we can start testing on test data.

You can think of evaluation data as a practice exam while test data is a real exam.

# make evaluation on evaluation data

# test on evaluation data
y_pred = model.predict(X_val_scaled)

# check the model's accuracy on evaluation data
accuracy = accuracy_score(y_val, y_pred)

# obtain precision, recall, and f1-score values
report = classification_report(y_val, y_pred)

# make evaluation on test data

# test on test data
y_pred = model.predict(X_test_scaled)

# check the model's accuracy on test data
accuracy = accuracy_score(y_test, y_pred)

# obtain precision, recall, and f1-score values
report = classification_report(y_test, y_pred)

We can write our own code for checking the accuracy above using the formula

The same goes for precision, recall, and F1-Score values.

:speech_balloon: FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.

Relevance: The audience of this lecture were those who have read Machine Learning 101 Warm-up, had some coding experience, and known a bit of AI knowledge.

In today’s post, we’ll explore K-Means and Unsupervised Learning. We’ll also discuss some applications in AI including tabular, image, and text data.

A dataset in Supervised Learning (SL) comes with two parts: X (the features) and Y (the labels). SL is used to solve classification and regression problems where we need the input data X and their results Y in order for a model to learn. In Unsupervised Learning (USL), a dataset only has X. Since there is no Y, USL cannot solve problems like classification or regression. Instead it can work on clustering problems which aim to group data points based on some property/condition; different properties/features can give different output.

Motivation

Let’s discuss a real-life problem. We have a dataset of sales from customers; this dataset does not give any information on customers, that is, is not labeled. The data is scattered and the whole dataset is not compactly represented.

The questions become, “How can we extract some useful information? More specifically, how can we classify customers into VIP and normal groups?”.

This is a typical example for the clustering problem.

For this particular question, we can think of the way and amount each group spends. The potential VIP tends to spend much more than a normal customer. Qualitatively, this can result in a clear divider between two groups. Visually, the customers are grouped into two sets which do not intersect/overlap with each other.

Nonetheless, to conclude it quantitatively, we need to have numbers to support our conjecture. Let’s experiment with a sample dataset.

:pushpin: Note: Even though the solution presented in the Sample works for a toy dataset, some problems needs addressing with this approach for it to work in real-world applications.

• Manually grouping and trying many possible of them for an actual dataset are very inefficient.
  • There is no clear way on how to make a grouping better.
• Unsystematically forming/guessing a grouping leads to a center biased by the grouping.
  • It’s hardly possible to get a proper center and its cluster.

To uplift the solution we discussed above, we can reverse the process by initially picking centers and then forming clusters.

$K$-Means

The improved version of the example above follows the exact procedure for grouping $n$ observations into $K$ clusters based on the distance in $K$-Means.

Procedure

We should always preprocess the data first. Then we can apply $K$-Means to a ML model in the following steps.

1. Initialize the value of $K$.
2. Select random $K$ centroid.
3. Assign each data point to the closest centroid.
4. Update a new centroid of each cluster.
5. If not convergent, go to step 3. Otherwise, stop.

flowchart LR
    A((<strong>Data processing</strong><br> select K</strong>)) ---> B((<strong>Select random<br> K centroids</strong>)) ---> C((<strong>Assign each data point<br> to the closest centroid</strong>)) ---> D((<strong>Update centroids</strong>)) --->|Not Convergent| E((<strong>If not convergent,<br> go to step 3</strong>))
    D --->|Convergent| F((<strong>Stop</strong>))

:pushpin: Note: $K$-Means always converges to a local minimum but does not guarantee convergence to global minimum (optimum). The result is dependent on the initialization.

$K$-Means Algorithm

You may question, “Why do we need to cluster data points? What do we do with them?

This question is on point. In USL, data is unlabeled, thus clustering data points into appropriate and suitable groups is the key to label the entire dataset based on the problem requirement.

%%{init: {'flowchart': {'curve': 'basis'}} }%%
flowchart TB
    %% Main Flow
    subgraph Algorithm ["Algorithm/Procedure"]
        direction TB

        subgraph IterateCentroids[" "]
            direction TB

            subgraph UpdateCentroid["Step 3: Update Centroids"]
                direction TB
                Recompute["Based on data points\n in the same centroid,\n re-compute the centroids"]:::dashedgreen
            end

            subgraph ClusterStep["Step 1: Find K Clusters"]
                direction BT
                RandomPickup["Random pickup"]:::dashed
                KCentroids1["K-Centroids"]:::core
                RandomPickup --> KCentroids1
            end
        end

        subgraph LabelStep["Step 2: Labeling data points"]
            direction LR
            AssignLabels["Assign the label of the nearest\n centroid for all the data points"]:::dashedorange
            NLabelled["N-data points (labelled)"]:::coreorange
            AssignLabels --> NLabelled
        end
    end

    subgraph Idea [Main Idea]
        direction TB
        Inputs["Inputs:\n• Unlabelled dataset\n• Number of Clusters (K)"]:::input
        Outputs["Outputs:\n• Centroids\n• Dataset labelled"]:::output
        Inputs ------> Outputs
    end

    NLabelled -.->|"Update centroids"| UpdateCentroid
    UpdateCentroid --> KCentroids1

    %% Cross connection
    KCentroids1 -.->|"Update labels"| AssignLabels

    %% Styling Definitions
    classDef input fill:#e6eef9,stroke:#7293ba,stroke-width:2px,font-family:serif;
    classDef output fill:#e6faeb,stroke:#628e6e,stroke-width:2px,font-family:serif;
    classDef dashed stroke:#7293ba,stroke-width:1px,stroke-dasharray: 6 4,fill:#f5faff,font-family:serif;
    classDef core fill:#c4e0fa,stroke:#3b6fa6,stroke-width:2px,font-weight:bold,font-family:serif;
    classDef dashedgreen fill:#f0faf4,stroke:#628e6e,stroke-width:2px,stroke-dasharray: 6 4,font-family:serif;
    classDef dashedorange fill:#faf2ea,stroke:#cd7a39,stroke-width:2px,stroke-dasharray: 6 4,font-family:serif;
    classDef coreorange fill:#fad1ae,stroke:#cd7a39,stroke-width:2px,font-weight:bold,font-family:serif;
    classDef labelHeader fill:#c4e0fa,stroke:#3b6fa6,stroke-width:2.5px,font-weight:bold,text-align:center;
    classDef orangeHeader fill:#ed8f33,stroke:#cd7a39,stroke-width:2.5px,font-weight:bold,color:#fff,text-align:center;
    class ClusterStep labelHeader
    class UpdateCentroid labelHeader
    class LabelStep orangeHeader

Formula

We can write these steps in mathematical expressions.

The goal of the clustering problem is to make the groups tight and compact while still covering all the data. Mathematically, it can be expressed as follows.

Tabular Data

Practice Exercises

Exercise 1: Use the following Iris data table and step-by-step $K$-Means procedure with Euclidean distance to label data points. Fill in the table below for each iteraton. Take $K = 2$, $c_0 = 1.4$, and $c_1 = 1$. Check the answer with code snippet below.

Index	Length	Distance to $c_0$	Distance to $c_1$	Label	New Centroids
0	1.4
1	1
2	1.5
3	3.1
4	3.8
5	4.1

Exercise 2: Do the same as exercise 1 and make suitable tables for the age-expenditure table with

1. $K = 2$, $c_0 = (18, 80)$ and $c_1 = (22, 85)$$
2. $K = 3$, $c_0 = (22, 85)$, $c_1 = (30, 50)$, and $c_2 = (34, 64)$

Exercise 3: Based on the code snippet for exercise 1 below and the following exercises, write code for exercise 2.
Exercise 4: What is the output of the following? Explain your answer.

data = np.random.rand(64,2)
centroids = np.random.rand(5,2)
distances = (data - centroids)**2

print(data.shape)
print(centroids.shape)
print(distances.shape)

data = np.random.rand(64,2)

data1 = data[:, np.newaxis, :]
data2 = data.reshape(1, 1, -1)

print(data1.shape)
print(data2.shape)

data = np.random.rand(34,4)
centroids = np.random.rand(3,4)
distances = np.sqrt((data[:, np.newaxis, :] - centroids)**2)

print(data.shape)
print(centroids.shape)
print(distances.shape)

Exercise 5: Which one gives the solution?

# 32 point, 3 dimensions (3 features), 4 centroids
squared_norm = np.random.rand(32,3,4)
# what is the distance between points and centroids?
np.sqrt(squared_norm.sum(axis=0))
np.sqrt(squared_norm.sum(axis=1))
np.sqrt(squared_norm.sum(axis=2))

# 32 point, 3 dimensions (3 features), 2 centroids
distances_detail = np.random.rand(32,3,2)
# what is the distance between points and centroids?
np.linalg.norm(distances_detail,axis=0)
np.linalg.norm(distances_detail,axis=1)
np.linalg.norm(distances_detail,axis=2)

Below is the code snippet for the practice exercise 1 just above.

import numpy as np

X = np.array([1.4, 1.0, 1.5, 3.1, 3.8, 4.1])

# step 1
k = 2

# step 2
centroids = X[:k]

X = X.reshape(6,1)

while True:
    # step 3a
    distances = np.sqrt((X - centroids)**2)
    print(f'distances:\n {distances}')

    # step 3b
    labels = np.argmin(D, axis=1)
    print(f'labels:\n {labels}')

    # step 4: update centroids
    new_centroids = np.array([X[L == i].mean() for i in range(k)])

    # step 5: check convergence
    if np.all(centroids == new_centroids):
        break

    # step 2: centroids
    centroids = new_centroids
    print(f'centroids:\n {centroids}')

We can also use an available package to solve the problem as well.

from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=2, max_iter=10)
kmeans.fit(X.reshape(6,1))
labels = kmeans.labels_
print(labels)
print(kmeans.inertia_)

for x, label in zip(X, labels):
    print(f'Cluster {label}: {x}')

In this example, our dataset consists of six data points $\{x_1, \cdots, x_6\}$ which are all one-dimensional real numbers, denoted $\mathbb{R^1}$ or $\mathbb{R}$. The initial $K = 2$ determines two clusters $\{S_0, S_1\}$.

Formula

Another critical obstacle when working with $K$-Means algorithm is to obtain the optimal $K$ or the optimal number of clusters. To do so, we need to determine how well the data points spread out from the centroids of their clusters.

The Within-Cluster Sum of Squares (WCSS) works like a signal for the best $K$ (like in the Sample example); it measures how compact the clusters are using a clustering procedure, making it easier to compare different $K$-clustering results.

The value of WCSS declines to 0 as $K$ grows. Using WCSS, we can figure out the optimal number of clusters via a method called the Elbow Method. This method aims to pick the elbow value where the rate of decrease for WCSS slows down sharply.

Image and Text Data

We can apply $K$-Means to image and text datasets as well. Before doing so, we must preprocess the data so that it becomes pertain to the algorithm.

Image Data

An image dataset can have many images. An image can be described as a matrix whose shape depends on its image type.

To treat this kind of dataset as input of $K$-Means algorithm, we first need to make each image into a vector just like a row/data point in the tabular data above. That is, we flatten matrices.

In practice, Deep Learning uses a magic function to process an image to extract its important parts which determine the content of the image. Just like our human brain, when we look at a cat or a dog image, we can tell what kind they are based on their face. The model completing this process is called a pretrained model. This reduces the size of data by a significant amount.

Text Data

Similar to image data, there is a magic function to learn important words which speak the main content of an article.

For text, we need to convert it into numeric values using vectorization algorithms before using $K$-Means.

:speech_balloon: FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.

Relevance: The audience of this lecture were those who have read Data Visualization and Analysis, had some coding experience, and known a bit of AI knowledge.

In today’s post, we’ll explore KNN and Supervised Learning. We’ll also discuss some applications in AI via a case study.

Machine Learning

Terminology

Machine Learning (ML) is a subfield of Artificial Intelligent (AI) focused on the study of algorithms that can imitate how humans learn and ultilize data to generalize or predict unseen information, and thus gradually improving their performance on specific tasks without explicit programming.

A familiar example is the classification problems we discussed in Math for AI series. Specifically, given a dataset with features and a class, we need to foresee a value of one column based on others. If a dataset is small enough, it may be possible to use multiple conditions to figure out. Nevertheless, it’s hardly possible for a large dataset with thousands and thousands lines of data. This is where ML comes into play.

In particular, we feed an ML model the dataset whose features and class values are complete and accurate. Using the given condition, the model takes note and learns to find out which function can match the input and output together. Behind the scenes, this model repeatedly tries out every possible features-class pair. Once it’s done, we need to check how well the model is doing.

The whole process includes two primary steps:

• training for learning data, requiring training data which is the mentioned dataset.
• testing for checking with non-dataset input, requiring test data.

The ML model after learning the entire dataset is called a trained ML model. With the testing phase, we can conclude how well a trained ML model works. This is known as machine learning generalization which refers to a train ML model’s ability to perform well on unseen data of similar types. For example, a model classifying dogs and cats should not be pedantic and distinguish African dogs and Asian dogs; they are just dogs anyway. We’ll talk more about this in the future.

Supervised Learning (SP)

Since this example of dataset has a class/label, we call it a labeled dataset. We call the type of machine learning where we use a labeled dataset to train a ML model a supervised learning (SL). This name comes from the idea that the data is set with specific class values.

A practical example is image classification for dogs and cats. The training data includes images of dogs and cats as a feature and a label column indicates which one is a cat and which one is a dog. The goal is for computers to answer if a test input image is a dog or a cat. Of course, if an input is a picture of a human being, the model will still result in a dog or a cat.

SL can solve various problems which can be categorized into two types: classification problems and regression problems.

Classification Problems
The class values in classification problems are discrete (categorical). Some examples can be dog-cat classification (0 and 1) and spam detection.

In general, for classification problems, we want to find a border line that best divide a dataset into separate regions/classes.

DEMONSTRATION

Regression Problems
The output predicted in regression problems are continuous—any numerical value falling within a certain range. Some examples can be predicting salaries and forecasting stock prices.

For regression problems, we generally want to find a line that best fits the data distribution.

DEMONSTRATION

$k$-Nearest Neighbors (KNN)

The core idea of KNN comes straight from the saying, “Show me your friends and I’ll tell you who you are.” That is, if you know someone’s circle, you’ll know what that person is like.

Formally, KNN is a non-parametric and SL algorithm which is used to make predictions based on proximity. In particular, the predicted class depends on the majority label / average value of its k-closest neighbors in the same feature space. This algorithm can be used for both classification problems and regression problems.

Some common metrics used to determine k-closet neighbors are listed in the table below. In the following formulae, $n$ is the number of features and $x_i$ and $y_i$ are coordinates of training and test points.

DEMONSTRATION for distances

Procedure

We should always preprocess the data first. Then we can apply it to a ML model in the following steps.

1. Initialize the value of $k$.
2. Compute the distance between test data and every training data (every row of the training dataset).
3. Sort the calculated distances in ascending order.
4. Get top $k$ elements from the sorted array.
5. Get the most frequent class of these elements.
6. Return the predicted class.

flowchart LR
    A((<strong>Initialize k</strong>)) ---> B((<strong>Compute distances</strong></br> between test data</br> and training data)) ---> C((<strong>Sort distances</strong></br> in ascending order)) ---> D((<strong>Get top k elements</strong></br> from the sorted array)) ---> E((<strong>Get the most frequent class</strong></br> of these elements)) ---> F((<strong>Return predicted class</strong>))

When working on step 3, keep in mind that we want to know the training points rather than just the distances. That is, we need to obtain the indices of training data.

The steps from 1 through 4 are quite straightforward while performing step 5 takes a bit more effort. This happens because of their class type differences. For classification problems, class values are discrete wheareas they are continuous for regression problems. That is, we need to approach two tasks via different strategies:

• Classification problems: discrete class set allows us to vote for the majority label from the top $k$.

  \[\hat{y} = max \\\{y_x | x \in NB \\\}\]

• Regression problems: continuous class set requires us to average the $k$-neighbors’ values.

  \[\hat{y} = \frac{1}{k} \sum_{x \in NB} y_x\]

:pushpin: Note:

• Results are potentially different depending on the initialization of $k$. In Machine Learning, parameters like k are referred to as hyperparameters—predictions change as hyperparameters change.
• If the number of labels equal each other, there are many ways to go about it. We can either increase $k$ or assign the label of the closet one.

Practice Exercises

Exercise 1: Use the following table and step-by-step procedure with Euclidean distance to check which label the test point $x\_test = 2.4$ should be when

1. $k = 1$
2. $k = 2$
3. $k = 3$
4. $k = 4$
5. $k = 5$

Exercise 2: Do the same as exercise 1 for $x\_test = (2.4, 0.8)$ and $x\_test = (2.6, 0.8)$ when

1. $k = 1$
2. $k = 3$
3. $k = 5$

Exercise 3: Write code for both exercises above.
Exercise 4: Use the following table and step-by-step procedure with Euclidean distance to predict the salary for $experience = 5.3$ when

1. $k = 2$
2. $k = 3$
3. $k = 4$

Exercise 5: Write code for exercise 4 above.

Scaling

As we’ve known so far, features and distances are the main contributors of KNN algorithm. In practice, if there is a problem with feature values, the distances will cause issues.

For example, if values of feature A are way greater than that of other features, the feature A can influence and dominate the results while it may not reflect its actual meaning/importance. This results in the incorrect learning of ML models—learning based on values solely rather than relevant meaning. It happens when we apply the procedure without any extra consideration.

To resolve this issue, we can build an algorithm that explicitly tells the model to focus on more significant features. Nevertheless, we do not want to make such suggestions every single time we work on a problem, which is quite a waste of resources. This is where scaling comes in handy.

Scaling is a crucial step in data preprocessing that normalizes and standardizes feature values (both training and test points) to fit in a certain domain. This helps the KNN algorithm focus on feature importance, ensuring feature values’ fair contributions to distance-based or gradiant-sensitive algorithm.

Choosing a domain depends on which scaling algorithm we use. Common domains can be $(0, 1)$ and $(-1, 1)$. One of the formula for normalization is the following

where $\bar{x}$ is the mean and $\sigma$ is the standard deviation of training data.

In practice, we use packages for scaling. One that is based on the above formula is

# import a package
from sklearn.preprocessing import StandardScaler

# call the function
scaler = StandardScaler()

# scale training and test values
x_data_scaled = scaler.fit_transform(x_data)
x_test_scaled = scaler.transform(x_test)

KNN for Classification Tasks

BoW is simpler and more basic approach to text representation than TF-IDF. BoW only considers word occurences while TF-IDF provides more informative representation of text—taking into account both word importance and context within the entire dataset. TF-IDF is generally preferred. We’ll delve deeper into the two techniques right away.

Bag-of-Words (BoW)

Problem statement: We have the following corpus.

Doc	Label
you reap what you sow	1
beggars can't be choosers	1
like bees to honey	1
bite the hands that feed you	0
a taste of your own medicine	0
burn your bridges behind you	0

Idea: BoW is to investigate distances between two sentences based on word occurences.

1. Tokenization splits text into a list of smaller parts or separate words.
2. Initialize an empty vocabulary. Add new words from tokenized corpus into the vocabulary.
Vocabulary =
3. Take a random string: "you can't pick and choose".
4. Convert the string to a numeric value by creating a BoW from the string and assigning a number to each of them. Two common ways are
• Binary BoW: Words in the string are assigned $1$ and others are $0$.
• (Complete) BoW: Each value is the number of its' word occurrency.
We use the (complete) BoW for this problem.
• From here, we build a matrix where each row is a word in the constructed vocabulary, each column is a sentence from the corpus, and each cell of a column is the BoW value of the corresponding sentence. Corpus sentences now become training vectors/points.
• With the random string, we can build a test vector/point.
DEMONSTRATION for matrix

Term Frequency-Inverse Document Frequency (TF-IDF)

Problem statement: We use the same corpus as in BoW. We can also use this approach to do spam email detection.
Breakdown:

• A document is a sentence/ an email.
• The corpus is a collection of documents.

Idea: TF-IDF is to compute the importance level of a word or phrase relative to its parent sentence/email and the entire corpus. Thus, TF-IDF helps identify words that are significant in a specific document but maybe rare across all the documents.

1. We build the same matrix for the entire corpus as in Bag-of-Words technique.
2. We build three other matrices:
• TF matrix: Each cell is the corresponding TF value of the row word and the column document.
• IDF matrix: Each cell is the corresponding IDF value of the row word and the column document.
• TF-IDF matrix: Each cell is the corresponding TF-IDF value of the row word and the column document.
• Normalized TF-IDF matrix: Each cell is the corresponding normalized TF-IDF value of the row word and the column document.
3. We build vectors for test documents.
4. Term Frequency (TF) measures how often a term/word appears in a document, that is how important a word is in a document.
$$\begin{align*} TF_{(t,d)} &= count(t,d) \\ &= \frac{\text{Number of times word$_\textbf{i}$ appears in a document}}{\text{Total number of words in the document}} \end{align*}$$ or in practice, $$TF_{(t,d)} = \log{(count(t,d) + 1)}$$ However, a word with a high occurence may not actually be of large importance, for example "the", "and", "are", and "is". We need additional step to handle this situation.
5. Inverse Document Frequency (IDF) measures how important a word is across the entire corpus.
$$\begin{align*} IDF_{t} &= \log \frac{\text{Total number of documents}}{\text{Number of documents containing word$_\textbf{i}$}} \\ &= \frac{N}{DF_t} \end{align*}$$ or in practice, $$\begin{align*} IDF_{t} &= \log \frac{\text{Total number of documents + 1}}{\text{Number of documents containing word$_\textbf{i}$ + 1}} + 1 \\ &= \frac{N + 1}{DF_t + 1} + 1 \end{align*}$$ Adding $1$ is to avoid $0$ as a result since it could kill of the importance of a word and avoid division by zero. This technique is called smoothing used in packages like scikit-learn.
6. TF-IDF computes the importance of a word in an email relative to the entire collection.
$$TF-IDF_{(t,d)} = TF_{(t,d)} \times IDF_{t}$$ This multiplication helps compensate for the disproportionate influence of frequently occuring but uninformative words and ensure the more meaningful features contribute appropriately to the distance calculations.
This next step is not required. The TF-IDF matrix contains many zeros and possiblly much greater values, so we need to normalize them like we perform scaling in KNN. This helps avoid bias from document length.
7. Normalized TF-IDF takes the unit vector of each TF-IDF vector. L2 norm is also known as Euclidean norm.
$$L2\_norm(TF-IDF) = \frac{TF-IDF}{\left \| TF-IDF \right \|_2} = \frac{\langle x_1, \cdots, x_n}{\left( \sum_i=1^n x_i^2\right)^\frac{1}{2}}$$ DEMONSTRATION for matrix

Case Study: Code Implementation

We primarily use the scikit-learn library for ML problems.

Problem statement: Build a system to predict whether a tumor is maglinant or benign based on real-valued information using KNN.

Set Up

# for a tabular dataset
import pandas as pd

# this dataset is available in the package, so we just need to import it
from sklearn.datasets import load_breast_cancer

# 
from sklearn.model_selection import train_test_split

# for scaling data (mentioned above)
from sklearn.preprocessing import StandardScaler

# KNN algorithm in the package
from sklearn.neighbors import KNeighborsClassifier

# for classification tasks and assessing how accurate our model is
from sklearn.metrics import accuracy_score, classification_report

The pipeline for a ML problem follows the basic procedure below. In practice, there can more steps to work through.

1. Data input
2. Data preprocessing
3. Data spliting:
   • Training data
   • Test data
4. Data training
5. Model evaluation

Data Input

# load and read the dataset
cancer = load_breast_cancer()

# convert the feature columns to a DataFrame
X = pd.DataFrame(cancer.data, columns=cancer.feature_names)

# convert the label column to a Series
y = pd.Series(cancer.target, name='target')

DEMONSTRATION FOR FIRST FIVE ROWS

Since the target contains non-numerical values: benign and malignant, we need to assign numeric values to them: $1$ for benign and $0$ for malignant.

Data Spliting
We mentioned splitting a dataset (label included) into two sets: training and test data. But in practice, a dataset is usally divided into three following sets:

• Training data
• Test data
• Evaluation data: we use this set to evaluate how accurate the ML model is—checking how close its predicted label values is to its actual label values, then check the same for test data.

val_size = 0.2
test_size = 0.125
random_state = 0
is_shuffle = True

# split training and test data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, 
    test_size=test_size, 
    shuffle=is_shuffle,
    random_state=random_state)

# split training and evaluation data
X_train, X_val, y_train, y_val = train_test_split(
    X, y, 
    test_size=val_size, 
    shuffle=is_shuffle,
    random_state=random_state)

Data Training

# scale data
# call scaling function
scaler = StandardScaler()

# scale both training and test data
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# initialize k
k = 5

# call KNN algorithm
knn_classifier = KNeighborsClassifier(n_neighbors=k)

# train data
knn_classifier.fit(X_train_scaled, y_train)

DEMONSTRATION IN SLIDE, ONLY TAKE 2 FOR VISUALIZATION, SHOULD BE 30 BUT THERE IS NO WAY TO VISUALIZE 30 DIMENSIONS

Model Evaluation
This step plays a very important role during the model development. Depending on the result, we can tune parameters, try different metrics, process the data differently or other methods. Once we get high-enough accuracy, we can start testing on test data.

You can think of evaluation data as a practice exam while test data is a real exam.

# make evaluation on evaluation data

# test on evaluation data
y_pred = model.predict(X_val_scaled)

# check the model's accuracy on evaluation data
accuracy = accuracy_score(y_val, y_pred)

# obtain precision, recall, and f1-score values
report = classification_report(y_val, y_pred)

# make evaluation on test data

# test on test data
y_pred = model.predict(X_test_scaled)

# check the model's accuracy on test data
accuracy = accuracy_score(y_test, y_pred)

# obtain precision, recall, and f1-score values
report = classification_report(y_test, y_pred)

We can write our own code for checking the accuracy above using the formula

The same goes for precision, recall, and F1-Score values.

KNN with K-D Tree

The original KNN algorithm has its own drawback, so methods like K-D Tree are introduced to improve the algorithm’s performance.

:speech_balloon: FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.

Relevance: The audience of this lecture were those who have read Data Visualization and Analysis for AI 101, had some coding experience, and known a bit of AI knowledge.

Visualizing data properly plays a significant role in learning how data is distributed and related before we delve into complex data analysis. In today’s post, we’ll explore common types of charts used in data visualization and more importantly, how to choose the right one for our dataset in general. We’ll look at real-world examples and case studies to see how different chart types work in action, giving more hands-on feel for the process.

Data Visualization

Data visualization is the process of transforming image, number, sound, or text information into visual elements like maps or graphs. In this way, we, as human users, can approach, comprehend, and identify insights from the information as a whole better and more easily.

If we have thousands and thousands of lines of values, it’d be extremely challenging to look at them and draw a conclusion. A chart made out of such a dataset can be much easier on the eye, not only for data experts but also regular people. For example, we wouldn’t know how president candidates are doing nationwide by just looking at numbers because of how Electoral College works.

Data visualization can be used on a wide range of data and take on various forms including 2Ds and 3Ds. Good data visualization helps lay the groundwork for further analysis like data mining, which ultimately supports smarter decision-making. There are different packages that we can use to visualize a dataset. In this post, we’ll focus on using three main ones:

• matplotlib: it is a quite complete visualization tool, offering extensive customization but requires a lot of parameters passed to a function, so users need to understand them more clearly.

  import matplotlib.pyplot as plt
  

• seaborn: it simplifies statiscal plots with built-in themes, focusing more on data itself rather than its meaning but makes it harder to customize the output.

  import seaborn as sns
  

• plotly: it provides dynamic and interactive visualization which users can engage with charts and see details without complicating charts.

  import plotly.express as px
  

It takes skill and experience to create a effective visualization. At its core, it all starts with the fundamental process. The figure below displays the full data science cycle—how visualization fits into the bigger picture.

flowchart LR
    A((Ask a question)) ---> B((Gather the data)) ---> C((Analyze the data)) ---> D((Convey the data)) ---> E((Communicate<br/>and Visualize<br/>the results))
    E ---> D ---> C ---> B ---> A
    E ----> A

:pushpin: Pro tip: Learning how to use libraries to visualize datasets is a small step in the big picture. Beginners should focus on building and sharpening critical thinking and analysis mindset to understand data and determine what charts to apply and how to use them.

Choose the Right Chart Type

There are five fundamental steps to go through when it comes to choosing the right chart type. Practically, we need to modify this process to be suitable for our datasets.

Determine the Type of Data

Data can be categorized into four following types with the first three being commonly encountered.

• Quantitative data: numerical values.
• Categorical data: non-numerical values, for example, text.
• Temporal data: numerical and time-series values.
• Spatial data: location-based values.

Identify the Relationship between Variables

A dataset can have hundreds of variables/properties which are stored in columns. When working with a dataset, we need to understand its properties and know which ones are valuable to problems raised. That is, we have to learn how they are linked with each other within the dataset. As such, we can narrow down the kinds of charts we should use to magnify their relationship.

There are four primary types of charts that corresponds to connection between variables.

• Comparison chart
  • used for showing differences
    • among multiple data
    • over time (to indicate trend)
  • a bar chart, column, or line chart.
• Distribution chart
  • used for showing how data is dispersed
  • a histogram or box plot.
• Relationship chart
  • used for showing how multiple properties are related
  • a scatter plot or bubble chart.
• Composition chart.

Determine the Purpose of Visualization

This is where asking questions also comes into play. This step can help us indicate what we really want from a dataset and what message we want to deliver through visualization.

Eventually, we want to answer the question how and what we want to show our data visualization.

• if we want to compare data points, a bar or column chart may be appropriate.
• if we want to show the distribution of data in specific properties a histogram or box plot might be useful.
• if we want to show a trend over time, we can use a line or area chart.

Identify the Audience

Just like public speaking, knowing our audiences—who we are creating charts for—is key when it comes to data visualization. The answer should guide how complex or simple our visuals need to be.

• For experts in the field, more advanced visuals like heat maps can be an adequate choice since these charts are equipped to interpret the details.
• For clients with little to none techincal background, keeping things simple like pie or bar charts is often more effective.

Select the Appropriate Chart Type

There’s no such thing as a one-size-fits-all solution to a data-related problem. Different probelms can have different visual solutions; it sometimes takes more than one chart to tell the full story clearly.

When we’re getting our feet wet, it’s imperative that we experiment various chart types. Play around, compare, and see which ones best present our data. Over time, as we become more experienced with different types of data and visuals, choosing the right chart will start to feel more intuitive and much quicker.

Case Study 1: Electricity Transformer Dataset (ETD)

Some abbreviations used in this dataset are in the following table.

First off, this dataset contains temporal data since it is in time series format. Let’s raise some relevant questions and see what charts might resolve them.

Line Chart

A line chart is used to demonstrate a single or multiple variables according to another variable. It is one of the most common charts utilized in trend analysis and time series analysis.

With a large dataset, it’s commonly challenging for audiences to see a specific part if creators only show an entire line chart. We can certainly make a zoom-in period on top of the main chart to highlight such an area.

DEMONSTRATION

Box Plot

A boxplot or box and whisker plot is used to portray the data outliers and distribution with minimum, maximum, and quartiles (excluding outliers) explicitly.

DEMONSTRATION lke

Quartiles are $25\%$, $50\%$, and $75\%$ we talked about in the previous post. We always sort the data before determining these quartiles.

• $50\%$ is the median of the entire dataset, denoted Q2.
• $25\%$ is the median of the first half, denoted Q1.
• $75\%$ is the median of the second half, denoted Q3.

An outlier is a data point that stands out—way out—from the rest. Outliers can have a big impact on how we interpret data. If we’re not careful, they can skew our results and lead us to the wrong conclusions. Take linear regression for example, a dataset with outliers can be pulled in a direction completely different from that without ones. Sometimes they are real valuable but can be just noise other times. Thus, it’s important to identify outliers and delicately handle them.

Data points that are not outliers are contained in an acceptable range. The formula for it is as follows.

where $IQR = Q3 - Q1$ is short for interquartile range.

This formula comes from the idea that outliers should stay outside of the range $(\mu - 3\sigma, \mu + 3\sigma)$ given that

• $(\mu - \sigma, \mu + \sigma)$ accounts for 68.27% data points.
• $(\mu - 2\sigma, \mu + 2\sigma)$ accounts for 95.45% data points.
• $(\mu - 3\sigma, \mu + 3\sigma)$ accounts for 99.73% data points.

Using these values, we can eventually have that $(\mu - 0.675\sigma, \mu + 0.675\sigma)$ accounts for IQR. If we try some numbers in the place of 1.5, we can conclude that it is actually what we want. This is called the $1.5$ IQR Rule created by John Tukey.

DEMONSTRATION outliers are small hollow circles above the max or below the min (excluding outliers).

Bar Chart

A bar chart is used to show the change in the values of a particular variable with respect to others or comparison among mutilple variables. It is also one of the most commonly used charts, generally appearing in discrete or categorical data. It can be horizontal or vertical.

DEMONSTRATION

:pushpin: Note: There are a couple of tips we can apply when working with data, especially for a bar chart.

• When there is a large difference among values of variables, it’s best practice to normalize the data to fit in a range uniformly.
• The stacked bar chart can be used as a filter to check the relibability of a dataset. That is, we can graph a stacked bar chart to see if there are too many negative/unacceptable values.

Donut Chart

A donut chart is a type of pie chart with a hole in the center, looking like a flat donut. This chart can be used to display the proportions of categorical data or check the reliability of a dataset like a stacked bar chart.

DEMONSTRATION

Correlation Chart

As its name suggests, a correlation chart is used to demonstrate the association between pairs of variables. There are two common types of correlation charts: a heatmap and a scatter plot.

A heatmap is basically a 2D colored table of squares showing the correlation coefficients between every pair of variables. Based on this chart, we can choose property pairs we’re interested in to further the investigation. However, this map is mainly for those who are familiar with data analysis.

We can instead use a pairplot to exhibit the pairwise bivaritate distrbutions to amateurs. Rather than coefficient numbers, it draws scatter plots’ all possible variable pairs in one chart. Using a pairplot may save time explaining what numerical values mean.

Taking one of the graphs from a pairplot, we obtain a scatter plot which is illustrate the correlationship between two specific properties.

DEMONSTRATION

Pie Chart

A pie chart shows the distribution of a dataset or relative contribution of each variable to the entire set.

DEMONSTRATION

Case Study 2: Student Performance Dataset (SPD)

Count Plot

A count plot is a column chart and basically used to count and show the number of observations in a dataset. It is very useful for categorical/non-numerical data points.

DEMONSTRATION

Case Study 3: Iris Dataset (IrD)

Histogram

A histogram is used to display the frequency distribution of a variable with horizontal or vertical bins. We came across this chart type when discussing image data.

We can apply a histogram to observe both discrete and continuous data. It can be considered as a detailed graph for a boxplot above. Looking at a histogram, we can tell how a dataset is spread out and how skewed it is.

DEMONSTRATION

Bubble Chart

A bubble chart looks similar to a scatter plot but for three variables. It is not a 3D plot; it subtlely corresponds the size of dots on the graph to the third variable.

DEMONSTRATION

KDE Chart

A Kernel Density Estimate (KDE) plot can be thought of as a smooth version of one or more histograms in one graph. Instead of using the real values, this plot visualize the probability density of a continuous variable. If there are at least two variables shown, the graph is called a 2D KDE plot. This chart can be found on the diagonal of a pairplot.

This chart is widely exploited in Machine Learning. We will talk more about the mathematical idea behind it in the future.

DEMONSTRATION

Displot Chart

A displot chart combines a histogram of a particular variable and the KDE, that is, a distribution curve on top. This can be use for both discrete and continuous data.

DEMONSTRATION

3D Visualization

A 3D plot is used to observe the relationship among three numerical variables. Some common 3D plots are scatter plots, surface plots, and contour plots.

For two numerical variables and one categorical variable, we just need to use a 2D plot with colors for each numerical values.

DEMONSTRATION

Case Study 4: App Review Dataset (ARD)

As we know, not all values in a dataset are numerical. For example, this dataset can have the “Rating” column include only numbers, but its “Review” column contains just words. How do we know which words are positive and which ones are negative?

Tag Cloud Chart

In this case, a common solution is to figure out the frequencey or importance of those words. A popular chart that can visualize this is called a tag cloud. It provides information about the context of the data. The bigger the word is, the more it appears in the dataset.

These kinds of visualization play a critical role in analyzing data from social networking websites and exploring data before performing Natural Language Processing (NLP).

DEMONSTRATION

Summary

Below are some tips for selecting the proper charts for visualization based on types.

:speech_balloon: FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.

Relevance: The audience of this lecture were those who have read Math for AI 106, had some coding experience, and known a bit of AI knowledge.

In today’s post, we’ll explore pandas Series and DataFrame. We’ll also discuss some applications in AI including processing missing values, noise reduction, and data visualization.

pandas was built on top of NumPy, specifically for data analysis and visualization, like Excel. Since it has NumPy as an underlying structure, we can use what we know about NumPy to apply here.

pandas Series

pandas Series is a 1D data array like a one-column table. Upone its creation, the numerical indices are automatically generated by default and shown in the rendered output, but they as a whole are not considered as a column. We can update the indices as desired—that is, an index can be any real number, a letter, or other data types.

Series Creation

Create a Series from a List

data1 = pd.Series([2, 5, 100, 7, 3, 4], name='score')
data1

'''
0      2
1      5
2    100
3      7
4      3
5      4
Name: score, dtype: object
'''

data2 = pd.Series(['C++', 'Java', 'Python', 'Golang', 'Swift'], 
                 index=list('CJPGS'),
                 name='Programming Language')
data2

'''
C       C++
J      Java
P    Python
G    Golang
S     Swift
Name: Programming Language, dtype: object
'''

Get Elements

Use data1. Just like ndarray in NumPy, we can apply indexing and slicing for pandas Series to get one or more elements.

# values in a Series is an ndarray
data1.values
>>> array([  2,   5, 100,   7,   3,   4], dtype=int64)

# indexing
data1[1]
>>> 5

# slicing
data1[2:4]
>>> 2   100
    3     7
    Name: score, dtype: object

A list of indices can be passed as an argument to obtain information from specific rows that may be unevenly-spaced.

data2[['C', 'J', 'C']]
>>> C     C++
    J    Java
    C     C++
    Name: Programming Language, dtype: object

pandas has other methods to obtain values in a Series like loc which may be more non-tech-friendly.

data1.loc[2:4]
>>> 2   100
    3     7
    4     3
    Name: score, dtype: object

Similarly, pandas also has boolean indexing.

# boolean indexing
data1[data1 < 4]
>>> 0    2
    4    3
    Name: score, dtype: int64

# boolean indexing
data1[data1.between(5, 7)]
>>> 1    5
    3    7
    Name: score, dtype: int64

For data2 with non-numeric indices, we can either use numeric or customized indices to retrieve elements.

data2['C']
>>> 'C++'

data2.loc['C':'J']
>>> C     C++
    J    Java
    Name: Programming Language, dtype: object

Operations

Drop a Row
We can remove an existing row by using drop() with indexing. This method returns a new Series instead of changing the original. Note that the returned Series keeps the original indices.

data1.drop([1, 3])
>>> 0      2
    2    100
    4      3
    5      4
    Name: score, dtype: int64

data1
>>> 0      2
    1      5
    2    100
    3      7
    4      3
    5      4
    Name: score, dtype: object

Insert a Row
Inserting a Row in pandas is similar to that for a Python dictionary. The new row is ed to the original Series or changes an existing row if they have the same index.

data2['K'] = 'Kotlin'
data2
>>> C       C++
    J      Java
    P    Python
    G    Golang
    S     Swift
    K    Kotlin
    Name: Programming Language, dtype: object

If an index is not an integer, we can certainly reset the Series so that it is properly indexed. This method does not change the original Series.

data1[2.5] = 99

# without drop=True, the old index column is ed
data1.reset_index(drop=True)
>>> 0      2
    1      5
    2      100
    3      7
    4      3
    5      4
    6      99
    Name: score, dtype: int64  

Sort a Series
Sorting a Series by its index can be done with method sort_index(). We can decide if the original Series should be updated or not by the argument inplace, in particular,

• inplace=True returns None and update the original Series.
• inplace=False returns a sorted Series and does not update the original Series.

# update the original Series
data2.sort_index(inplace=True)
data2
>>> C       C++
    G    Golang
    J      Java
    K    Kotlin
    P    Python
    S     Swift
    Name: Programming Language, dtype: object

# NOT update the original Series
sorted_data2 = data2.sort_index(inplace=False)
sorted_data2 
>>> C       C++
    G    Golang
    J      Java
    K    Kotlin
    P    Python
    S     Swift
    Name: Programming Language, dtype: object

data2
>>> C       C++
    J      Java
    P    Python
    G    Golang
    S     Swift
    K    Kotlin
    Name: Programming Language, dtype: object

Common Functions
pandas contains NumPy’s common functions

• series.min()
• series.max()
• series.idxmin() or series.argmin()
• series.idxmax() or series.argmax()
• series.sum()
• series.mean()
• series.var()
• series.std()
• series.str.count(): the argument can be a character or a string and it counts the number of characters or strings in each value.
• series.str.upper()
• series.replace(current_value, new_value): replace the current value with a new value.

** Two Series**
If values from Series are of the same data type, they can be ed or concatenated. Behind the scenes, pandas looks for the matching index and two values together. If it cannot find a row’s matching index, it will render NaN.

# modify data1 to have 6 rows
data1[2.5] = 99
data1.sort_index(inplace=True)

# create data3 of 5 rows
data3 = pd.Series([10, 10, 10, 10, 10])

#  data1 and data3
data1 + data3
>>> 0.0     12.0
    1.0     15.0
    2.0    110.0
    2.5      NaN
    3.0     17.0
    4.0     13.0
    5.0      NaN
dtype: float64

Group Values
If some values have the same name/index and we want to group them together, we can use groupby(). This syntax returns another Series, so we can certainly use Series methods or function on it.

data4 = pd.Series([4, 1, 0, 2, 2, 1, 8],
                  index=list(['C++', 'Golang', 'Java', 'Golang', 'Java', 'C++', 'Python']),
                  name='groupby')

# test
data4.groupby(level=0).sum()
>>> C++         5
    Golang      3
    Java        2
    Python      8
    Name: groupby, dtype: int64

data4.groupby(data4 > 3).sum()
>>> groupby
    False     6
    True     12
    Name: groupby, dtype: int64

data4['Golang'].count()
>>> 2

:bulb: Application in AI

1D Interpolation

1. The problem introduced here appears in scaling images and the two techniques are a part of Super-resolution Imaging. Since scaling an image increases the number of cells but newly ed cells need values.
2. Another situation is when we work with missing data points. We can either remove or keep them depending on the type of datasets. For those that stay, we need to make sure they have values. For example, time series data can be related with each other, so we’d like to keep missing values.

In both cases, we need to fill in context-appropriate rather than random values. For some datasets, we fill in with the average. In ition, we use interpolate() with a method argument to resolve this issue.

• Linear Interpolation: method='linear' by default.
• Nearest Neighbor Interpolation: method='nearest'.

data5 = pd.Series([2, 5, 100, np.nan, 7, 3, np.nan, 9, np.nan], name='score')

# linear
data5.interpolate(method='linear')

# nearest neighbor
data5.interpolate(method='nearest')

In the example above, pandas considered 100 and 7 for the first np.nan.

Nearest Neighbor Interpolation vs. Linear Interpolation

Problem statement: Assume there are two data points, find another data point given one of its coordinates.
Breakdown:

• $x$ is the index, $y$ is the value in a Series.
• $(x_1, y_1)$ and $(x_2, y_2)$: two given data points.
• $x_3$: known coordinate of the third data point.
• $y_3$: unknown coordinate of the third data point.

Idea: There are two simple but effective techniques: nearest neighbor interpolation and linear interpolation.

	Nearest Neighbor	Interpolation
Solution	Compute distance from $x_3$ to $x_1$ and $x_2$. Pick the y-value of the smaller one.	Find the linear equation passing $(x_1, y_1)$ and $(x_2, y_2)$. Take the corresponding y-value on that line. $$\begin{align*} &y_3 = \frac{x_2 - x_3}{x_2 - x_1} y_1 \\ &+ \frac{x_3 - x_1}{x_2 - x_1} y_2 \end{align*}$$
Application	If objects in an image are straight up perpendicular to the edges, they should look fine when the image is scaled. If they are slanted, they could look like a floor function, Lego blocks, or images in Minecraft.	This can be better for slanted objects.

Noise Reduction
Noise refers to the random irregularity or fluctuations in real-life datasets. There are a few ways to reduce noise in statistics.

Simple Moving Average (SMA): the simplest way in which we take the mean of data windows.

where $k$ is the size of each window.

Exponential Moving/Weighted Average (EMA): widely used in optimization, considering both the previously computed values and the current value.

Later on, we’ll mainly use EMA, but for now let’s look at the graph of the above dataset after using SMA. The larger the window is, the smoother the graph is.

linear_sma_5 = linear_int.rolling(window=5).mean()
linear_sma_5.head(100).plot()

DataFrame

A DataFrame is a 2D labeled data structure. Unlike a Series, a DateFrame can have more than one data column. Basically, it is a combination of multiple Series. Thus, we can apply methods/functions in Series to work with DataFrames.

Get Values
We can also apply indexing and slicing for pandas Series to get one or more elements like a NumPy array—using a comma to separate axes.

df = pd.read_csv('advertising_simple.csv')
df
>>>     TV  Radio   Newspaper   Sales
    0   44     39          45      10
    1   17     45          69      12
    2   151    41          58      16
    3   180    10          58      17
    4   8      48          75      7
    5   57     32          23      11
    6   120    19          11      13
    7   8      2           1       4

# get 'Radio' column
df.loc[:, 'Radio']
>>> 0     39
    1     45
    2     41
    3     10
    4     48
    5     32
    6     19
    7      2
    Name: Radio, dtype: int64

# get a DataFrame of 'Radio' column
df.loc[:, ['Radio']]
>>>    Radio
    0     39
    1     45
    2     41
    3     10
    4     48
    5     32
    6     19
    7      2

# another way to get 'Radio' column
df.Radio
>>> 0     39
    1     45
    2     41
    3     10
    4     48
    5     32
    6     19
    7      2
    Name: Radio, dtype: int64

Besides loc, pandas has iloc to get values using index rather than label. Similar to NumPy, pandas has boolean indexing.

df.loc[df.Sales > 10]
>>>     TV  Radio   Newspaper   Sales
    1   17     45          69      12
    2   151    41          58      16
    3   180    10          58      17
    5   57     32          23      11
    6   120    19          11      13

Sort Values
pandas lets us sort data based on one or multiple columns sequentially.

# sort by one column
df.sort_values(['Newspaper'])

# sort by more than one column
# 'Newspaper' first, then based on that, 'Sales'
df.sort_values(['Newspaper', 'Sales'])

Inser a Column
We can insert a column to a DataFrame like ing an item in Python dictionary. The original DataFrame is updated. Besides the default index ‘colum’, we can certainly set a column to be indices. Note that setting indices does not affect the original.

#  'ID' column
df['ID'] = list(range(2, 10))

# set index
df = df.set_index('ID')
df
>>>     TV  Radio   Newspaper   Sales
    ID				
    2   44     39          45      10
    3   17     45          69      12
    4   151    41          58      16
    5   180    10          58      17
    6   8      48          75      7
    7   57     32          23      11
    8   120    19          11      13
    9   8      2           1       4

Concatenate Rows
If we want to combine two DataFrames with the same columns together, we use concat([dataFrame1, dataFrame2]). Since the original indices from each DataFrame remain unchanged, we may need to reset index and drop the old one by reset_index(drop=True).

Datetime
When data is collected, time is often inserted as strings, which makes it harder for computers to understand and analyze. Thus, it’s recommended time data be converted to real datetime format. Use to_datetime() method.

df = pd.read_csv('WeatherHistory2D.csv')

# convert string to datetime format
time_col = 'Formatted Date'
df[time_col] = pd.to_datetime(df[time_col],
                              utc=True)

# get each feature of datetime 
#  a column for each feature
df['Year'] = df[time_col].dt.year
df['Month'] = df[time_col].dt.month
df['Day'] = df[time_col].dt.day
df['Hour'] = df[time_col].dt.hour

Remove a Column
In the example above, the “Formatted Date” column is no longer needed as we separate columns for datetime features. We can just delete it using df.drop(label, axis). This syntax returns a new DataFrame rather than changing the original. We can certainly delete multiple columns.

df.drop(time_col, axis=1)
>>>     Temperature (C) Year    Month   Day Hour
    0      0.577778     2005       12    31   23
    1      1.161111     2006       1     1    0

Group Values
We can certain group values by columns. That is, if there are duplicate rows in a column, we can combine them together and use functions to analyze each of them as a whole.

df_group = pd.read_csv('weatherHistory_simple.csv')
df_group
>>>     Precip Type Temperature (C)
    0          rain            12.2
    1          rain            13.8
    2          rain            14.7
    3          rain            13.8
    4          rain            12.7
    5          rain            0.5
    6          snow            -0.4
    7          snow            -1.1
    8          snow            -1.6
    9          snow            -2.1

# group by 'Precip Type' column and compute mean of each feature by 'Temperature (C)' column
df_group.groupby('Precip Type').mean('Temperature (C)')
>>>         Temperature (C)
Precip Type	
rain              11.283333
snow              -1.300000

Convert Categories to Numbers
In the example above, the “Precip Type” column can be a class column as in a classification problem. However, computers can only work with numeric values, it’s better that we convert those string categories into numbers.

# test
col_to_num = 'Precip Type'
pd.Categorical(df_group[col_to_num])
>>> [0, 0, 0, 0, 0, 0, 1, 1, 1, 1]
    Categories (2, int8): [0, 1]

pd.Categorical(df_group[col_to_num]).codes
>>> array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1], dtype=int8)

# convert categories to numbers
col_to_num = 'Precip Type'
df_group[col_to_num] = pd.Categorical(df_group[col_to_num]).codes
df_group
>>>     Precip Type Temperature (C)
    0             1           12.2
    1             1           13.8
    2             1           14.7
    3             1           13.8
    4             1           12.7
    5             1           0.5
    6             0           -0.4
    7             0           -1.1
    8             0           -1.6
    9             0           -2.1

Descriptive Statistics
We can check out all the basic information related to a dataset by info(). pandas also calculates basic descriptive statistics and lets us see them via describe() method.

df_weather.loc[:, ['Temperature (C)']].describe()
>>>         Temperature (C)
    count      96453.000000
    mean       11.932678
    std        9.551546
    min        -21.822222
    25%        4.688889
    50%        12.000000
    75%        18.838889
    max        39.905556

The numbers $25\%$, $50\%$, and $75\%$ are called k-th percentiles where $k$ is the percentage.

• $50\%$ is the median.
• $25\%$ means one fourth of the sorted dataset from minimum.
• $75\%$ means one fourth of the sorted dataset from maximum.

Using percentiles, we can indicate if the distribution of a dataset is more left-skewed or right-skewed.

:pushpin: Pro tip: pandas includes many other functions and methods that allows us to analyze how training models work for a dataset. This helps us know which model we should use for our datasets in certain cases.

Visualization

We can use seaborn, which is built on top of matplotlib to visualize data.

# set up seaborn
import seaborn as sns

We’ll use a famous Iris flower classification example here.

df_iris = pd.read_csv('iris.csv')

# graph two columns in one figure
sns.jointplot(df_iris, x='PetalLength', y='PetalWidth', hue='Name')

This kind of plot is quite beneficial for both classification and prediction problems when we analyze to choose the right columns to compare.

We can use pairplot() to see all the pairs.

We can also check out the distribution of each type as follows. Boxplot gives the statistical structure of a feature where min, three primary percentiles, and max are shown from top to bottom by default.

# dotplot 
sns.stripplot(df_iris, y='SepalLength', x='Name', hue='Name')

# boxplot
sns.boxplot(df_iris, y='SepalLength', x='Name', hue='Name', width=0.3)

:speech_balloon: FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.

Relevance: The audience of this lecture were those who have read Math for AI 102, had some coding experience, and known a bit of AI knowledge. Absolute beginners are encouraged to read the material and repeatedly practice coding without AI assistance to strengthen their coding and code-reading skills. Note that only necessary pandas knowledge is introduced here.

In today’s post, we’ll explore basic pandas using examples: Pokemon Data Analysis and Time Series Dataset.

pandas

Introduction

Data analysis is a process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making. That is, it includes visualization like charts, graphs, or maps based on which users can produce insights about a dataset.

pandas is a library built on NumPy written specifically for Python language. It is a data manipulation and analysis tool, providing simple and practical data structures ready for usage.

Check out pandas documentation for more packages. We can set up pandas and check its version as follows.

!pip install pandas

import pandas as pd

pd.__version__
>>> 

There are three primary data structures in pandas, but one of them is barely used these days, so we’ll focus on the other two: DataFrame and Series. Since pandas is built on NumPy, its data structures have certain similarities with ndarray.

DataFrame

A DataFrame is a 2D labeled data structure in which data in columns can be of different types. Visually, it is like 2D tables with headers in Excel.

Upon its creation, indices for rows are automatically generated by default. But we can certainly set up our own indices.

We are allowed to create a dataframe from various data structures such as arrays, dictionaries, lists, zip() functions, or even an existing DataFrame. Take a look at an example below.

data = {
    'Name': ['Brucy', 'Fiona', 'Ziggy', 'Ranger'],
    'Gender': ['M', 'F', 'F', 'M']
}

df = pd.DataFrame(data)
df

'''
    Name    Gender
0   Brucy   M
1   Fiona   F
2   Ziggy   F
3   Ranger  M
'''

Series

A Series is a 1D labeled array that can hold any data type. Upon its creation, indices for rows also are automatically generated by default. But we can explicitly name the indices. Since there is only one non-indexed column, the header row is not shown in the rendered table, but we can display it by converting the Series to a DataFrame.

data = np.array(['Brucy', 'Fiona', 'Ziggy', 'Ranger'])

s = pd.Series(data, name = 'Name')
s

'''
0     Brucy
1     Fiona
2     Ziggy
3    Ranger
Name: Name, dtype: object
'''

The definitions of both data structures infer that a DataFrame is essentially a combination of a Series sequence. Continue from the DataFrame example.

name_series = df['Name']
type(name_series)
<class 'pandas.core.series.Series'>

gender_series = df['Gender']
type(gender_series)
<class 'pandas.core.series.Series'>

new_df = pd.concat([name_series, gender_series], axis=1)
type(new_df)
<class 'pandas.core.frame.DataFrame'>

Thus, we can also create a DataFrame from Series.

# create two Series
name_series = pd.Series(['Brucy', 'Fiona', 'Ziggy', 'Ranger'], name='Name')
gender_series = pd.Series(['M', 'F', 'F', 'M'], name='Gender')

# create DataFrame from Series
df_from_series = pd.concat([name_series, gender_series], axis=1)
df_from_series

'''
    Name    Gender
0   Brucy   M
1   Fiona   F
2   Ziggy   F
3   Ranger  M
'''

In the next two sections, we’ll use a couple of hands-on examples to learn pandas step by step. These datasets are written in .csv files, which are quite common for storing data.

Pokemon Data Analysis

Read and Save a CSV File
If we print data out, we’ll see the information table and its numbers of rows and columns.

DATASET_PATH = 'Pokemon.csv'
data = pd.read_csv(DATASET_PATH)

We can certainly save a dataset to a file under various format. The following is for a .csv file.

data.to_csv('new_Pokemon.csv')

Head/Tail Method
With this method, we can check out how many first and last rows we want with some following syntax. By default, five rows at the top or at the bottom of the DataFrame will be shown when no argument is added.

data.head()
data.head(n=7)
data.head(900)
data.tail(10)

Rows
Not only header/tail rows, we can read an equally-spaced sequence of rows by slicing a DataFrame using [start:stop:step].

data[2:10:3]

'''
    #   Name                        Type 1  Type 2  Total   HP  Attack  Defense Sp. Atk Sp. Def Speed   Generation  Legendary
2   3   Venusaur                    Grass   Poison  525     80  82      83      100     100     80      1           False
5   5   Charmeleon                  Fire    NaN     405     58  64      58      80      65      80      1           False
8   6   CharizardMega Charizard Y   Fire    Flying  634     78  104     78      159     115     100     1           False
'''

However, we cannot do indexing to read one specific row. Instead, we can use iloc—index-based selecting method. We can treat a row index as an array index or a real row in a table. Thus, the datatypes of two ways are different—one is a Series and the other is a DataFrame.

Only a snippet of the output is provided in the following code.

# this renders a Series
data.iloc[7]

'''
#                                     6
Name          CharizardMega Charizard X
Type 1                             Fire
...                                 ...

Name: 7, dtype: object
'''

A Series is like a 1D array, so we can get its elements by their indices or slicing. Or you can think of it as getting an element from a 2D array where the first argument is index/slice for rows and the second for columns.

data.iloc[7, 10]

'''
100
'''

# this renders a DataFrame
data.iloc[[7]]

'''
    #   Name                        Type 1  Type 2  Total   HP  Attack  Defense Sp. Atk Sp. Def SpeeD   Generation  Legendary
7   6   CharizardMega Charizard X   Fire    Dragon  634     78  130     111     130     85      100     1           False
'''

Since it is a DataFrame, it can handle multiple rows and columns simultaneously. We can read more rows by adding more row indices to the list.

# this renders a DataFrame
data.iloc[[7, 10]]

'''
    #   Name	                    Type 1  Type 2  Total   HP  Attack  Defense Sp. Atk Sp. Def Speed   Generation  Legendary
7   6   CharizardMega Charizard X   Fire    Dragon  634	    78  130	    111	    130	    85	    100	    1	        False
10  8   Wartortle                   Water   NaN	    405	    59  63	    80	    65	    80	    58	    1	        False
'''

If row indices are renamed to something esle, we can use loc—label-based selecting method to check out rows. Because of how it works, we must use a header or a list of headers rather than an index or slicing to get elements.

data.loc[7, 'Legendary']

'''
False
'''

This is also very powerful since it can filtering rows. We’ll talk more about it later when we have sufficient tools.

Reset Index
As we see above, indices of rows are kept the same as in the original dataset when extracted. This is inconvenient to work with just that part of the dataset. We can save it to a new variable and reset index. The original indices are kept and the new indices is added.

new_slice = data.loc[7:20:3]
new_slice.reset_index()

'''
    index   #   Name                        Type 1  Type 2  Total   HP  Attack  Defense Sp. Atk Sp. Def Speed   Generation  Legendary
0   7       6   CharizardMega Charizard X   Fire    Dragon  634     78  130     111     130     85      100     1           False
1   10      8   Wartortle                   Water   NaN     405     59  63      80      65      80      58      1           False
2   13      10  Caterpie                    Bug     NaN     195     45  30      35      20      20      45      1           False
3   16      13  Weedle                      Bug     Poison  195     40  35      30      20      20      50      1           False
4   19      15  BeedrillMega Beedrill       Bug     Poison  495     65  150     40      15      80      145     1           False
'''

Headers
We can see all the headers of columns in a dataset. They will

data.columns
>>> Index(['#', 'Name', 'Type 1', 'Type 2', 'Total', 'HP', 'Attack', 'Defense',
           'Sp. Atk', 'Sp. Def', 'Speed', 'Generation', 'Legendary'],
           dtype='object')

Columns
There are two simple ways to check out data values in particular columns. We can treat a header as an index of a list or a real header of a table. Like rows above, one result is a Series and the other is a DataFrame.

Only a snippet of the output is provided in the following code. Check out the practice exercise to read more than one columns.

data['Name']

data[['Name']]

Row Filtering
Like an array, we can filter rows via comparison operations. We can see whole rows or just a few columns of satisfied rows.

First, let’s apply loc.

# only rows with data value being True in the Legendary column are selected
data.loc[data['Legendary'] == True]

# only a column and rows with data value being True in the Legendary column are selected
data.loc[data['Legendary'] == True, 'Name']

data.loc[data['Legendary'] == True, ['Name']]

data.loc[(data['Type 1'] == 'Grass') & (data['Type 2'] == 'Poison')]

In addition, we can choose selective rows by Regex with package import re.

import re

# rows with name containing 'Mega'
data.loc[data['Name'].str.contains('Mega')]

# rows with type 1 being either 'Fire' or 'Grass'
data.loc[data['Type 1'].str.contains('Fire|Grass')]

Describe Method
pandas provides basic descriptive statistics on every numerical column of a dataset with a simple syntax describe().

data.describe()

Arranging Order
We can sort a dataset based on columns and the result is in an increasing order by default. We can add the ascending=False or ascending=0 argument to reverse it. We use a list to sort multiple columns and various orders. This method does not affect the original data.

# sort by one column
data.sort_values('Total')

# sort by multiple columns
data.sort_values(['Total', 'Type 1'], ascending=[1, 0])

We can also arrange columns in a certain order. This produces a DataFrame of selected columns in the desired order. Thus, this method does not affect the original data.

data[['Type 1', 'Name']]

Update Data
We can add or remove columns for a dataset. Use drop() for column removal; this does not affect the original dataset. On the other hand, adding a column updates the original dataset. If an existing column is modified, it will be updated rather than a new column is added.

# remove a column and store the new dataset to a new variable
new_data = data.drop(columns='Total')

# add a column
# add two numerical columns
new_data['Attack + Defense'] = new_data['Attack'] + new_data['Defense']

groupby Function
We can group values of columns based on their types.

data.groupby(['Type 1', 'Type 2']).count()

'''
                    #   Name    Total   HP  Attack  Defense Sp. Atk Sp. Def Speed   Generation  Legendary
Type 1	Type 2											
Bug     Electric    2   2       2       2   2       2       2       2       2       2           2
        Fighting    2   2       2       2   2       2       2       2       2       2           2
...     ...         ... ...     ...     ... ...     ...     ...     ...     ...     ...         ...  

Water   Ice         3   3       3       3   3       3       3       3       3       3           3
        Poison      3   3       3       3   3       3       3       3       3       3           3
'''

We can further this step by using an additional function to manipulate data. You can check out more functions as needed.

data.groupby(['Type 1', 'Type 2'])[['HP']].apply(
    lambda x: x.sum() / len(x)
)

Time Series Dataset

A time series dataset including a continuous time period.

ts_df

'''
        Date        Daily minimum temperatures
0       1/1/1981    20.7
1       1/2/1981    17.9
2       1/3/1981    18.8
3       1/4/1981    14.6
4       1/5/1981    15.8
...	    ...	        ...

3645    12/27/1990  14
3646    12/28/1990  13.6
3647    12/29/1990  13.5
3648    12/30/1990  15.7
3649    12/31/1990  13
'''

Time Format Conversion
A dataset may have a date or time column, but their values might not be in the right format. pandas allows us to convert such data to time format which makes accessing a certain date/time row convenient. We can also get each part of a date/time; for example, day, month, or year.

# time format conversion
ts_df['Date'] = pd.to_datetime(ts_df['Date'])

# set 'Date' column as an index column
ts_df = ts_df.set_index('Date')

We can use the code below with some modification to create new meaningful columns for time.

# index_cols=0 takes the first column
# parse_dates reads data as time, not string
df = pd.read_csv('timeseries_daily-minimum-temperatures.csv', index_col=0, parse_dates=True)

df.index.year
df.index.month
df.index.day_name()

Time-based Indexing
As mentioned, once we convert the ‘Date’ column to time format, we can retrieve rows via time-based indices.

# get rows from one date to another by slicing
ts_df.loc['1987-08-29':'1997-03-30']

# get ros in a specific month
ts_df.loc['1990-01']

Time Series Data Visualization

pandas
In order to visualize data, values should be numeric for computers to understand. If there are values that cannot be converted into a number, for example, due to wrong format, we can either remove such a row or force the value to be fixed. Otherwise, an error is produced.

col_to_plot = 'Daily minimum temperatures'

# force wrong values to be fixed
ts_df[col_to_plot] = pd.to_numeric(ts_df[col_to_plot], errors='coerce')

# plot
ts_df[col_to_plot].plot(
    linewidth=0.5,
    figsize=(10,4),
    legend=True
)

# plot
ts_df[col_to_plot].plot(
    marker='.',
    alpha=0.5,
    linestyle='None',
    figsize=(10,4),
    legend=True
)

Seasonality with seaborn
When dealing with time series datasets, it’s important to understand data values according to their time blocks. This is called seasonality. Formally, seasonality is a characterisitc of a time series where data experiences roughly periodic changes recurring yearly.

seaborn is a Python data visualization library built on top of matplotlib.

import matplotlib.pyplot as plt
import seaborn as sns

# set figure size
sns.set(rc={'figure.figsize':(10,4)})
# plt.figure(figsize=(10,4))

# boxplot
sns.boxplot(ts_df, x='Month', y=col_to_plot)

Frequencies
With time format, we can get a sequence of time periods based on their frequency and/or the number of periods. Frequency is day by default, but it can be, say hour.

pd.date_range('1997-03-30', '1997-05-03')

pd.date_range('1997-03-30', '1997-05-03', freq='h')

pd.date_range('1997-03-30', periods=8, freq='h')

Assume that we have only 3 random time series data points. But we want to convert data to daily frequency.

# convert string to time format
times_sample = pd.to_datetime(['1981-01-01', '1981-01-04', '1981-01-08'])

# make a DataFrame to manipulate
consum_sample = ts_df.loc[times_sample, ['Daily minimum temperatures']].copy()
consum_sample

# convert data to daily frequency
# new rows has no data
consum_freq = consum_sample.asfreq('D')
consum_freq

# create a column with missings to be forward filled
# forward fill is with 'before' data
# backward fill is with 'after' data
consum_freq['Daily minimum temperatures - Forward Fill'] = consum_sample.asfreq(
    'D', method='ffill'
)
consum_freq

'''
            Daily minimum temperatures
1981-01-01  20.7
1981-01-04  14.6
1981-01-08  17.4


            Daily minimum temperatures
1981-01-01  20.7
1981-01-02  NaN
1981-01-03  NaN
1981-01-04  14.6
1981-01-05  NaN
1981-01-06  NaN
1981-01-07  NaN
1981-01-08  17.4

            Daily minimum temperatures  Daily minimum temperatures - Forward Fill
1981-01-01  20.7                        20.7
1981-01-02  NaN                         20.7
1981-01-03  NaN                         20.7
1981-01-04  14.6                        14.6
1981-01-05  NaN                         14.6
1981-01-06  NaN                         14.6
1981-01-07  NaN                         14.6
1981-01-08  17.4                        17.4
'''

Resampling
Resampling is a technique used when values in each data block are close with each other. For example, temperatures at 11am, 12am, and 1pm are 99, 100, 101 degree Fahrenheit. We can somehow take only one value to represent for these data points to save resources.

• Find a common feature of values and use it to calculate a common number. Take mean or median for example.
• If there is no common feature, use one of the values.

col_to_resample = 'Daily minimum temperatures'

# take the mean of temperatures by week
ts_weekly_mean = ts_df[[col_to_resample]].resample('W').mean()
ts_weekly_mean.head(3)

'''
            Daily minimum temperatures
Date	
1981-01-04  18.000000
1981-01-11  17.542857
1981-01-18  20.371429
'''

# plot daily and weekly resampling for a period
start, end = '1981-01', '1981-12'

# create the plot
fig, ax = plt.subplots()

ax.plot(
    ts_df.loc[start: end, col_to_resample],
    marker='.',
    linestyle='-',
    linewidth=0.5,
    label='Daily Minimum Temperatures'
)

ax.plot(
    ts_weekly_mean.loc[start: end, col_to_resample],
    marker='o',
    linestyle='-',
    label='Weekly Mean Resampling'
)

ax.set_ylabel('Temperature')
ax.legend()

:speech_balloon: FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.

Relevance: The audience of this lecture were those who have read Math for AI 105 or been familiar with probability and statistics. Note that only basic probability material is introduced here; we’ll pick up specific concepts when in need.

In today’s post, we’ll delve into some more descriptive statistics. We’ll also discuss some applications in AI including template matching.

Measure of Central Tendency

We previously talked about measure of central tendency in general and only briefly mentioned population and sample. We’ll discuss more in detail to distinguish the two concepts.

Population vs Sample: Mean, Variance, and Standard Deviation

Why not $n$?
The sample variance calculated by dividing by $n$ is often referred to as the biased/naive sample variance. These name refer to a major problem it causes.

The problem is that we would consistently underestimate the variance around the population mean. This happens because, on average across all possible samples drawn from the population, the naive sample variance is smaller than the population variance. That is, there is a disparity between the two

1. Let’s first understand what each side of this equality means.
   • The right-hand side is simply the population variance.
   \[\frac{1}{N} \sum (x_k - \mu)^2 = \sigma^2\]
   • The left-hand side is the expected value of all possible naive sample variances, particularly the mean in this case. We have the left-hand side can be rewritten as
   \[E \left( \frac{1}{n} \sum_{k=1}^{n} (x_k - \bar{x})^2 \right) = E(\tilde{s}^2).\]

   Here, we are not just comparing a single sample’s naive variance. Instead, we want to analyze the relation between the naive sample variance and the entire population variance overall, that is, from a global point of view. Our goal is to drag the sample variance to be close to the population variance as much as possible.

2. Before working out the detail, let’t see how this idea might work in the case of a single sample’s naive variance and why we need the average.

   • Claim: \(\sum_{i=1}^{n} (x_k - \bar{x})^2 < \sum_{i=1}^{N} (x_k - \mu)^2 \text{ for any } \bar{x}\).

   • Proof: According the claim, we want to minimize the sum of squared deviations. The common method is to figure out the absolute minimum, in particular local minimum in this case.

     Consider $f(a) = \sum_{k=1}^{n} (x_k - z)^2$. Differentiating the function, we have

     \[\begin{align*} \frac{df}{dz} = \frac{d}{dz} \sum_{k=1}^{n} (x_k - z)^2 = -2 \sum_{k=1}^{n} (x_k - z). \end{align*}\]

     The local minimum of the function is at

     \[\begin{align*} &-2 \sum_{k=1}^{n} (x_k - z) = 0 \Rightarrow \sum_{k=1}^{n} (x_k - z) = 0 \\ &\Rightarrow \sum_{k=1}^{n} x_k = \sum_{k=1}^{n} z = nz \Rightarrow z = \frac{\sum_{k=1}^{n} x_k}{n} = \bar{x}. \end{align*}\]

     This proves our claim, which suggests there might be a bias if we divide by $n$. So the denominator cannot be $n$.

     The graphs below display the idea using different sample sizes. As the sample size increases to the population size, the sample mean will get closer to the population mean.

We cannot conclude the claim for the variance since the denominators are different. However, we know that the denominator of the naive sample variance should be smaller. It can be $n - 1$, $n - 2$, $n - 3$, and so on.

Why not $n - 2$, $n - 3$, and so on?

1. Note: This is a proof of our main claim. You can read the next part if you think it may be pedantic.

   Since we want to compare with the population variance, we want to involve $\mu$ in the left side of \eqref{naive-inequality-problem}.

   \[\begin{align} (x_k - \bar{x})^2 &= ((x_k - \mu) - (\bar{x} - \mu))^2 \nonumber \\ &= (x_k - \mu)^2 - 2(x_k - \mu)(\bar{x} - \mu) + (\bar{x} - \mu)^2. \label{bar_x-to-mu} \end{align}\]

   Using \eqref{bar_x-to-mu}, we have

   \[\begin{align} \sum (x_k - \bar{x})^2 &= \sum (x_k - \mu)^2 - \sum 2(x_k - \mu)(\bar{x} - \mu) + \sum (\bar{x} - \mu)^2 \nonumber \\ &= \sum (x_k - \mu)^2 - 2(\bar{x} - \mu) \sum (x_k - \mu) + n (\bar{x} - \mu)^2. \label{sum-bar_x-to-mu} \end{align}\]

   We know that

   \[\begin{align} \label{x_k-mu} \sum (x_k - \mu) = \sum x_k - \sum \mu = n \bar{x} - n \mu = n (\bar{x} - \mu) \end{align}\]

   Substitute \eqref{x_k-mu} into \eqref{sum-bar_x-to-mu}. We have

   \[\begin{align} \sum (x_k - \bar{x})^2 &= \sum (x_k - \mu)^2 - 2(\bar{x} - \mu) n (\bar{x} - \mu) + n (\bar{x} - \mu)^2 \nonumber \\ &= \sum (x_k - \mu)^2 - 2n (\bar{x} - \mu)^2 + n (\bar{x} - \mu)^2 \nonumber \\ &= \sum (x_k - \mu)^2 - n (\bar{x} - \mu)^2. \label{result-bar_x-to-mu} \end{align}\]

   Substituting \eqref{result-bar_x-to-mu} into the left side of \eqref{naive-inequality-problem}, we have

   \[\begin{align} E \left( \frac{1}{n} \sum (x_k - \bar{x})^2 \right) &= E \left(\frac{1}{n} \left[\sum (x_k - \mu)^2 - n (\bar{x} - \mu)^2 \right] \right) \nonumber \\ &= E \left(\frac{1}{n} \left[\sum (x_k - \mu)^2 \right] \right) - E \left(\frac{1}{n} n (\bar{x} - \mu)^2 \right) \nonumber \\ &= \frac{1}{n} \sum [E ((x_k - \mu)^2)] - E ((\bar{x} - \mu)^2) \nonumber \\ &= \frac{1}{n} n [E ((x_k - \mu)^2)] - E ((\bar{x} - \mu)^2) \nonumber \\ &= [E ((x_k - \mu)^2)] - E ((\bar{x} - \mu)^2) \label{expectation-naive-sample-variance} \end{align}\]

   Assume all the samples are independent and identically distributed—having the same variance $\sigma^2$. By definition of the general variance and linear combination rule, we have

   \[\begin{align} \label{same-variance} E ((x_k - \mu)^2) = var(x_k) = \sigma^2 \end{align}\]

   and

   \[\begin{align} E ((\bar{x} - \mu)^2) &= var(\bar{X}) = var \left(\frac{1}{n} \sum x_k \right) = \frac{1}{n^2} \sum var (x_k) \nonumber \\ &= \frac{1}{n^2} \sum \sigma^2 = \frac{1}{n^2} n \sigma^2 = \frac{\sigma^2}{n}. \label{variance-of-sample-mean} \end{align}\]

   Replacing \eqref{same-variance} and \eqref{variance-of-sample-mean} into \eqref{expectation-naive-sample-variance}, we have

   \[\begin{align} \label{result-expectation-naive-sample-variance} E(\tilde{s}^2) = E ((\bar{x} - \mu)^2) = \sigma^2 - \frac{\sigma^2}{n} = \frac{n - 1}{n} \sigma^2. \end{align}\]

   Thus, \eqref{result-expectation-naive-sample-variance} indicates that $\tilde{s}^2$ gives an estimate $\frac{n - 1}{n} \sigma^2$ of the population variance $\sigma^2$. This estimation contributes to the bias or difference by a factor of $\frac{n - 1}{n}$ between the two variances claimed in \eqref{naive-inequality-problem}.

2. To correct this bias, we need to eliminate that exact factor by multiplying by its reciprocal. The result after correction is called (unbiased) sample variance, denoted $s^2$ as in the formula.

   \[s^2 = \frac{n}{n - 1} \tilde{s}^2 = \frac{n}{n - 1} \left( \frac{1}{n} \sum_{k=1}^{n} (x_k - \bar{x})^2 \right) = \frac{1}{n - 1} \sum_{k=1}^{n} (x_k - \bar{x})^2\]

Degrees of Freedom

In statistics, the degrees of freedom specify the number of independent variables used in the calculation of statstic—any quantity computed from values in a sample. In laymen terms, it is the number of information pieces required to know the rest of a sample which is not free to vary.

Think about $y = a_1x_1 + \cdots a_nx_n$. This equation involves $n$ independent variables, say $x_1, \cdots, x_n$, since the value of $y$ depends on them and can be inferred knowing their values. So the degree of freedom is n.

Let’s say if you toss a coin, there are two possible outcomes: head and tail. If you say it is not a tail, then it must be a head. This means head can be derived from the information on tail—if tail is independent, head is not. So there is 1 degree of freedom. The same goes for doing this experiment many times.

Another example is that if we know the sample mean and all values but one in a sample of size $n$. The unknown is completely determined by the $n - 1$ known values because the mean must stay fixed. So the sample has $n - 1$ degrees of freedom. This is another reason why the unbiased estimator of variance divides by $n - 1$ as the constraints are all we need to control the sample space.

Measure of Variability

Covariance

In the previous post, we talked about variance, which shows how dispersed the data values are from the center (mean). That’s an individual measurement.

If we want to understand the relationship between more than one single set of data, we need covariance. Covariance measures the linear relationship of two random variables, that is, how they change together.

DEOMONSTRAION

Even though covariance have quite a few disadvantages, it is a major computational stepping stone for the development of advanced statistical methods like correlation coefficient or principal component analysis.

Correlation

Correlation measures the strength and direction of a linear relationship between two variables. Basically, it describes how two variables are linked or associated with each other. Correlation does not mean one variable causes the other; there could be a third variable influencing both of them and thus this co-relationship.

Correlation can be expressed by either a scatter plot or a correlation coefficient. The table below shows characteristics of Pearson’s correlation coefficient. We’ll talk more about it in the next part as well.

DEOMONSTRAION

Correlation vs Regression
Consider $y = ax + b$. In this equation, $x$ is an independent variable while $y$ is dependent, that is, $y$ is affected/determined by $x$. This idea describe regression of two variables. Formally, regression describes a functional relationship between two variables—how one is dependent of the other. With such a property, we can use it to estimate the unknowns and make predictions based on assumptions.

• Correlation may imply regression since we can model a regression line but it might not be a good one.
• Regression can indicate correlation since regression how much one variable depends on the other.

Correlation vs Causation
Causation indicates the cause and effect relationship between two variables.

• Correlation does not imply causation.
• Causation may imply correlation since that one variable causes the other means we can expect some correlation. Due to various factors, the correlation might not be clear.

The figure below illustrates a high correlation between icecream sales and swimming activity throughout the year. But it is wrong to conclude the sales is the reason for swimming activity or vice versa.

In practice, it can sometimes be unethical to force researches on causality, so researchers tend to use correlation to conclude if there is a relationship between events. Take smoking and lung cancer or domestic violence and mental illness for example.

Regression vs Causation

• Regression does not imply causation.
• Causation only implies regression if the relationship is linear and quantifiable.

Correlation Coefficient

Some common correlation coefficients are Pearson’s r, Spearman’s rho, Point-biserial, Cramer’s V (Cramer’s $\varphi$), and Kendall’s tau. We’ll only go over Pearson’r in this post.

Earlier, we noted that Pearson’s r is insensitive to scaled data. This mathematically means

as

and

Computers often represent correlation as a 2-by-2 matrix.

Practice Exercises

Exercise 1: Use a package in NumPy to compute the covariance between sets of data in the picture.

Exercise 2: Prove the three equations that show Pearson'r is sensitive to scaled data.
Exercise 3: Write a function to compute Pearson's r in four ways (expected value, data points, NumPy package, and Scipy package) and test it with X = [7, 18, 29, 2, 10, 9, 9] and Y = [1, 6, 12, 8, 6, 21, 10].
Exercise 4: For exercise 3, what is noteworthy about the results using pacakges?
Exercise 5: What can you say if two random variables have correlation being 1?
Hints: Property of correlation coefficient.

:bulb: Applications in AI

Traffic Sign Matching

In Math for AI 102, we discussed some image matching applications. We can use two following concepts to check which images in a given set are most similar.

• absolute difference / distance
• cosine similarity

In addition, we can apply the correlation concept to handle the same situation. It is not only insensitive to scaled data in \eqref{corr-coeff-scale-invariant} like cosine similarity but invariant to brightness shift in \eqref{corr-coeff-brightness-shift-invariant} as well. This linear change insusceptibility in \eqref{corr-coeff-linear-change-invariant} facilitates correlation coefficient to perform well on the same images of different brightness.

DEOMONSTRAION

Image Processing: Template Matching

Template matching is a technique used to find a portion of an image that matches a template image.

We can use all three concepts to resolve this kind of problem. But we know absolute difference and cosine similarity both have more shortcomings compared to correlation coefficient which we’ll use here.

1. It’s recommended to work with grayscale version of images.
2. Since we already have a template, we can consider it as a sliding window to run all over the input image.
3. At each step, we can compute and store the correlation coefficient between the template and the checking portion in a correlation map.
4. The highest value give our matching.
5. Take the coordinates of the matching portion to highlight it on the original image.

We can also visualize the correlation map by representing it with colors.

# template matching

import cv2
import numpy as np
from matplotlib import pyplot as plt

# Load image and convert to grayscale
image = cv2.imread('image.png',1)
gray  = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

template = cv2.imread('template.png',0)
w, h = template.shape[::-1]

# Apply template Matching
corr_map = cv2.matchTemplate(gray, template, cv2.TM_CCOEFF_NORMED)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(corr_map) 

# take minimum
top_left = max_loc
bottom_right = (top_left[0] + w, top_left[1] + h)

# draw a bounding box on the input image
cv2.rectangle(image, top_left, bottom_right, (0, 255, 0), 2)

# save corr map in grayscale
corr_map = (corr_map+1.0)*127.5
corr_map = corr_map.astype('uint8') 
cv2.imwrite('corr_map_grayscale.png', corr_map)

# applyColorMap
corr_map = cv2.applyColorMap(corr_map, cv2.COLORMAP_JET)

# save results
cv2.imwrite('corr_map_color.png', corr_map)
cv2.imwrite('result.png', image)

:speech_balloon: FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.

Relevance: The audience of this lecture were those who have read Math for AI 104 or been familiar with probability and statistics. Note that only basic probability material is introduced here; we’ll pick up specific concepts when in need.

In today’s post, we’ll explore some descriptive statistics and probability distribution theory. We’ll also discuss some applications in AI including image processing.

Descriptive Statistics refers to tools that summarize and describe features of a dataset, providing us with a clearer picture of the data. There are three main categories: central tendency, variability, and distribution.

Measure of Central Tendency

Mean and Expected Value

Expected value is not considered a measure of central tendency, but this concept and mean are closely related as they both represent some sort of average.

Median

Given a dataset of N items $X = \{X_1, \cdots, X_N\}$, median is the value dividing the sorted dataset in half. To obtain the median, we follow two steps.

1. Sort $X$.
2. Calculate the median $m$:
   • If $N$ is odd, then $m = X_{\frac{N + 1}{2}}$.
   • If $N$ is even, then $m = \frac{1}{2} (X_{\frac{N}{2}} + X_{\frac{N + 1}{2}})$.

:bulb: Applications in AI

Image Processing: Image Blurring using Mean
The media show pictures or videos and some images are wished to be blurred for privacy or security. We can apply a process called image blurring to accomplish this purpose.

Using a average/mean filter (AF) is a common and simple method to execute this process. The core idea behind it is to smooth out sharp edges and reduce the intensity variations by replacing each pixel’s value with the average of its neighboring pixels including itself. Averaging pixel values is akin to mixing paint colors togther. The more colors we mix, the more the distinctiveness of individual colors is lost.

Correlation ~ Convolution
To implement this idea, we use a technique called correlation ~ convolution. We know image data is an ndarray or a matrix where each appropriate set of values is a pixel.

Take a grayscale image for example. This can be turned into a 2D $m \times n$ matrix. We want to blur the whole image. The code is below.

1. Determine the shape of neighboring pixel areas, say a $j \times k$ matrix.
2. Take a matrix of that shape and call it a kernel.
3. Elements of this matrix are valued $\frac{1}{\text{number of elements}} = \frac{1}{j \cdot k}$ since we want to compute mean.
4. Slide the kernel over each neighboring pixel area from left to right and from top to bottom to obtain those pixels’ mean.
5. Place the mean on the position of the top-left-corner pixel. It’s best practice to store those means in a new array.
6. The new matrix’s shape is $(m - j + 1) \times (n - k + 1)$.

The change in shape makes the new image smaller. To handle this issue, we need a technique calld padding—adding layers of numbers surrounding the original matrix. This will take care of the lost region. Here we can

• add $j - 1$ and $k - 1$ layers of zeros for the first and second dimension, respectively.
• copy edge values to the padding region.

Of course, we can also blur part of an image. This part is called Region of Interest (ROI). Recall that an image is an array part of which we can take out by slicing. Check out Math for AI 101 for more information about updating a slice of an array.

• Static image: If we already know the coordinates of ROI, using slicing with specific numbers is suitable.
• Static and moving image: If we don’t know the exact coordinates of ROI, we need to detect that part first and then apply slicing. For a facial area, we can use the Haar-Cascade Face Detection algorithm, which returns the top-left-corner coordinates, height, and width of ROI. The face detection set up is as follows.

face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

For color images, we can compute mean for each channel and concatenate three channels in the end to rebuild the third dimension.

There are some available packages we can use to apply put this technique into practice.

• NumPy: np.einsum()
• Scipy: spicy.signal.convolve2d()
• OpenCV: cv2.filter2D().

DEMONSTRATION MAYBE VIDEO IF TOO HARD TO WRITE

# load image and blurring

import numpy as np
import cv2

# load image in grayscale mode
image = cv2.imread('mrbean.jpg', 0)

# create kernel
kernel = np.ones((5,5), np.float32) / 25.0

# compute mean for each pixel
dst = cv2.filter2D(image, cv2.CV_8U, kernel)

# show images
cv2.imshow('image', image)
cv2.imshow('dst', dst)

# waiting for any keys pressed and close windows
cv2.waitKey(0)
cv2.destroyAllWindows()

Image Preprocessing: Image Denoising using Median
Pictures are often not taken by professionals and in the studio condition. They that can be blurred, underexposed or of unwanted speckles are usually referred to as grainy or noisy images. Hanlding these problems which is a part of image preprocessing is called image denoising. Using median is one of the simpliest method to address this.

One common type of noise is salt-and-pepper noise, that is, an image is speckly. We’ll go over an example of this type.

This median filter (MF) also uses the correlation ~ convolution technique to obtain the median of neighboring pixels. We can use package cv2.medianBlur() for this purpose.

Using median rather than mean in this case is because getting median can exclude too-large or too-small values which are noises. Using mean only makes those values smaller or larger but still keeps noises. The new image is also blurred after median filter but not as much affected as using mean filter.

# denoise salt and pepper

import numpy as np
import cv2

img1 = cv2.imread('mrbean_noise.jpg')

# compute median in each kernel with side=3
img2 = cv2.medianBlur(img1, 3)

# show images
cv2.imshow('img1', img1)
cv2.imshow('img2', img2)

# waiting for any keys pressed and close windows
cv2.waitKey(0)
cv2.destroyAllWindows()

Measure of Variability

Variance and Standard Deviation

:bulb: Application in AI

Image Processing: Image Texture using Standard Deviation

When identifying an object, computers may just need to know its shape. That is, we focus on the boundary of objects in an image rather than their color. This is known as image texture—local patterns created by differences in relative color arrangement in an image. We will discuss this concept in detail later in this AI series.

Since we’re only interested in the edge of objects in an image, it suffices to just work with a grayscale version of an image for the sake of simplicity. This standard deviation filter also uses the correlation ~ convolution technique to obtain the standard deviation of neighboring pixels.

• The standard deviation remains small when the kernel moves over regions with little color variation or uniform intensity.
• The standard deviation increases sharply when the kernel moves across a boundary from a consistent-color region to a region with contrasting colors.

# extract image texture

import numpy as np
import cv2
import math
from scipy.ndimage.filters import generic_filter

# convert a color image to grayscale
img  = cv2.imread('img.jpg')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
cv2.imwrite('edge_s1.jpg', gray)

# compute standard deviation in each kernel of size=7
x = gray.astype('float')
x_filt = generic_filter(x, np.std, size=7)
cv2.imwrite('edge_s2.jpg', x_filt)

# reassign small values (no sudden change in standard deviation)
# blacken insignificant areas or edges
x_filt[x_filt < 20] = 0
cv2.imwrite('edge_s3.jpg', x_filt)

# lighten significant edges
x_filt = x_filt*2.5
cv2.imwrite('edge_s4.jpg', x_filt)

Probability Distribution Function

A probability distribution is a function that describes the likelihood of occurence of possible events, that is, random variables. It is used to model the behavior of random phenomena from a dataset. There are three main types of probability distribution and we’ll discuss them in a moment.

• Probability Mass Function (PMF)
• Probability Density Function (PDF)
• Cumulative Distribution Function (CDF)

Discrete: PMF and CDF

PMF measures the probability of a given single value x occuring in a countable dataset, denoted by $p_X(x) = P(X = x)$. CDF measures the probability of values occuring from the beginning up to x in a countable dataset, denoted by $F_X(x) = P(X \le x)$.

Take rolling a dice for example. We have $p(3) = \frac{1}{6}$ and $F(3) = p(1) + p(2) + p(3) = \frac{3}{6}$.

Continuous: PDF and CDF

For experiments like assessing grades for an examination, there are so many outcomes that the probability to take on any particular value is always 0. We can hardly use PMF to understand the likelihood of a single value occuring in a dataset. This is where PDF comes into play.

PDF measures the relative likelihood of a random variable to take a value near x, denoted by $f(x)$. It is used to specify the probability that a given single value x falls within an interval, computed by integrate PDF $\int_{a}^{b} f(x) \,dx$. CDF measures the probability of values occuring from the beginning up to x in an uncountable dataset, denoted by $F_X(x) = P(X \le x)$.

Gaussian distribution function \(f(x) = \frac{1}{\sigma \sqrt{2\pi}} \exp^{-\frac{(x-\mu)^2}{2\sigma^2}}\) is a commonly used PDF. The total area under this curve is always 1 and

• the area within 1 𝜎, that is, $[\mu - \sigma, \mu + \sigma]$ is about 62.27%.
• the area within 2 𝜎, that is, $[\mu - 2\sigma, \mu + 2\sigma]$ is about 95.45%.
• the area within 3 𝜎, that is, $[\mu - 3\sigma, \mu + 3\sigma]$ is about 99.73%.

EXAMPLE DEMONSTRATION

:bulb: Application in AI

Measure of Frequency Distribution: Histogram
Given a grayscale image, we can convert it into an array or matrix. Using such information, we can extract and arrange values in a frequency histogram. Such a histogram represents the frequency distribution of pixel values from the image. Pixel values range from 0 to 255 and we call each value a bin, so we have 256 bins in total. This allows us to imply if a grayscale image is bright or dark without even seeing the image as values a pixel gets lighter and lighter as it goes up.

The output of a frequency histogram depends on the number of pixels in an image as well as each bin. Thus, the output varies among images. We can normalize the output so that it acts as the probability density. This results in a density histogram.

Let $N$ be the total number of pixels of an image and $n_b$ be the number of pixels in bin $b$. The density value of bin b is

So the (probability) cumulative value up to bin b is computed as follows. We can use these values to construct a cumulative histogram—displaying how pixel values accumulate across the intensity range.

In fact, using any of these histograms allows us to determine the brightness of an image.

3 DEMONSTRATION from slide 54 describe how and why they are darker or brighter in each image caption

Histogram Equalization
Besides determining how bright or dark an image is, we can even change the brightness of an image by utilizing the probability distribution functions.

The core idea behind contrast adjustment is to modifying the distribution of the density histogram of an image so that the cumulative histogram approximates to a linear relation. That is, we wish uniformly redistribute the values of the frequency or density histogram. This helps the resulting image has a uniform intensity distribution. This process is known as histogram equalization.

Essentially, the ideal scenario is when all the bins have the same number of values. This gives a straight line when accumulating those values. It’s like a dice roll experiment.

DEMONSTRATION

We do not use density histogram here because it only describes individual bins—there is no relation between intensity values. Instead, to achieve this purpose, we use the cumulative histogram, which tells us how pixel intensities build up across the image. The formula that we’ll explore soon maps the original pixel intensity $b$ to a new value proportional to $H[b]$.

• A very dark pixel like 5 may be pushed to a brighter intensity.
• The accumulation levels out the brightness by spreading values uniformly.
  • If over two thirds of pixels are valued in [0, 50], they’ll be mapped to high intensity values like [0, 200].
  • And only a quarter of the rest falls in [50, 100], they’ll be mapped to just slightly higher intensity values or a narrow range like [200, 225].

  This way, no bin would get way too much addition than others.

Let $L$ be the number of gray/intensity levels. It corresponds to the number of possible values a pixel can take within an image. Each pixel in this case ranges from 0 to 255, so there can be 256 different gray levels.

Note that the cumulative values are in [0, 1], but the gray values are in [0, 255]. This is the reason to employ the multiplication and floor function.

We can also apply a similar method for a color image.

Example with a 1D Array for Simplicity

Problem statement: Given the following matrix. Demonstrate histograms before and after the histogram equalization process. $$\begin{bmatrix} 1 & 2 & 7 & 5 & 6 \\ 7 & 2 & 3 & 4 & 5 \\ 0 & 1 & 5 & 7 & 3 \\ 1 & 2 & 5 & 6 & 7 \\ 6 & 1 & 0 & 3 & 4 \end{bmatrix}$$ Breakdown:

• A table contains all the calculations above.
• N = 25 since it is a 5-by-5 matrix.
• L = 8 since the values run from 0 to 7.
• Show original and equalized histograms to see how new values change the graphs.

Idea: We apply the histogram equalization process.

$b$	Frequency	PMF ~ $h[b]$	CDF ~ $H[b]$	$(L - 1) \cdot H[b]$	$b_{new}$
0	2	0.08	0.08	0.56	1
1	4	0.16	0.24	1.68	2
2	3	0.12	0.36	2.52	3
3	3	0.12	0.48	3.36	3
4	2	0.08	0.56	3.92	4
5	4	0.16	0.72	5.04	5
6	3	0.12	0.84	5.88	6
7	4	0.16	1	7.0	7

:speech_balloon: FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.

Relevance: The audience of this lecture were those who have read Math for AI 104 or been familiar with probability and statistics. Note that only basic probability material is introduced here; we’ll pick up specific concepts when in need.

In today’s post, we’ll explore basic probability concepts: random variables, probability distribution, and descriptive statistics.

Random Variable (RV)

Definition

Let’s use some beginner-friendly scenarios to introduce the idea of a random variable.

From above cases, we can think of a random variable as a real number depending on the outcome of a random experiment.

Formally, a random variable $X$ is a function $X: \Omega \to \mathbb{R}$, $s \mapsto X(s)$ where $\Omega$ is the sample space—set of outcomes and $\mathbb{R}$ is the set of real numbers. Let’s delve into two types of random variables and their examples to nail this concept.

Discrete RV - Probability

A discrete random variable $X$ is a function $X: \Omega \to \mathbb{R}$ where $\Omega$ is a finite or countably infinite set. Since the range of this function contains numbers representing mutually exclusive events, we can measure probability on random variables and the sum of all those must be 1.

Probability on random variables are known as probability distribution. Probability mass and density are common probability distributions.

Probability Mass Function (PMF), denoted by $p(x)$ or $p_X(x)$, measures the probabiltiy of possible values of a discrete random variable, that is, $p(x) = p_X(x) = P(X = x)$.

Cummulative Distribution Function (CDF), denoted by $F(x)$ or $F_X(x)$, is the cumulative probability of of all possible values less than equal to a specific number, that is, $F_X(x) = P(X \le x)$.

We’ll see examples demonstrated in two following tables.

Below, consider the number of heads in the sequence as a random variable.

Continuous RV - Probability

A continuous random variable $X$ is a function $X: \Omega \to \mathbb{R}$ where $\Omega$ is an uncountabe set. The normal distribution mentioned in Math for AI 104 is one of the most well-known probability models for continuous random variables .

Let’s see a couple of examples.

1. Imagine we’re looking at the second hand of a clock and take a random snapshot at an unknown moment during a one-minute period.
   • Define $X(0, 12]$ = angle $\theta$ (in degrees) between the second hand and the 12 o’clock position.
   • The second hand sweeps smoothly in a full circle ($360^\circ$) every minute.
   • So $X \in (0, 360]$.
2. Adult female/male heights.
   • A person’s height can be any positive real number.
   • So $X \in (0, \infty)$.

PMF for continuous random variables is always 0 since the sample space is so large that it makes the occurence of one value become insignificant. On the other hand, PDF for continuous random variables can be computed using the Gaussian function.

Mathematical Expectations

Below are some concepts used for discrete random variables. We’ll briefly go over mathematical expectations for continous random variables right after.

Expected Value

Expected value of a discrete random variable X, denoted by $E[X]$, represents the weighted average value of X and computed as follows. It is often referred to as the mean of probability distribution but tremendously more useful when the size of the sample space is unknown.

Some important properties of expected values are

• $E[\alpha X] = \alpha E[X]$
• $E[\alpha X + b] = \alpha E[X] + b$.

Variance

Variance of a random variable X, denoted as $var(X)$, is the weighted average of squared distance from the mean. The reason behind squared distance is that we want to variance to be non-negative for further statistics concepts. On top of that, squared is preferred over absolute value since such a squared function is differentiable everywhere. Use np.var().

where $\mu = E[X]$.

Standard Deviation

Standard deviation of a random variable X, denoted $\sigma(X)$, is the weighted average of distance from the mean. Use np.std().

:pushpin: Note: (discussed more in Math for AI 106)

1. If outcomes are equally likely to happen and $N$ is the entire population’s size, we have

   \[E[X] = \frac{1}{N} \sum_{k} x_k \text{ and } var(X) = \frac{1}{N} \sum_{k} (x_k - \mu)^2.\]

2. If we only take a sample of size $n$, then N becomes $n - 1$. Add another argument ddof=1 when using NumPy.

Continuous Random Variable

Since PMFs for continuous random variables are always 0, we can only apply the idea to compute expected value, variance, and standard deviations rather than exactly the same formulas. Here, we use integral for expected value which is

This integral can be done using either Riemann’s sum or Lebesgue integral (and maybe other integrals). The former is applied for “nice” distribution while the latter can handle more complicated distributions. This is partly because the Lebesgue integral takes weighted coefficients into account better. Computers use the Lebesgue integral as its default method.

Covariance and Correlation

:question: We’ve been dealing with cases with only one random variable. What if we’d like to know the relationship between two random variables?

This is where covariance and correlation come into play.

Covariance

Covariance measures the linear relationship of two random variables, that is, how they change together. Recall $n - 1$ is for samples as we consider $X$ and $Y$ as subset of the population. Use np.cov().

where $X$ and $Y$ are random variables such that $E[X] = \mu_X$ and $E[Y] = \mu_Y$.

We deduce from the above formula that $var(X) = cov(X, X)$. Formally, covariance is represented using the following 2-by-2 matrix

We can interpret the result as follows.

• $cov(X, Y) \in (-\infty, \infty)$.
• $cov(X, Y) = cov(Y, X)$.
• If it is positive covariance or $cov(X, Y) > 0$, then $X$ increases implies $Y$’s rise on average.
• If it is negative covariance or $cov(X, Y) < 0$, then $X$ reduces implies $Y$’s decrease on average.
• If it is zero covariance or $cov(X, Y) = 0$, then there is no linear correlation between two random variables.

DEMONATRATION

Correlation

Even though covariance is useful, its value can be unnecessarily large. In addition, covariance only indicates the direction of the relationship, not how much it is. To address this issue, we standardize the covariance to obtain the correlation coefficient.

Correlation is a statistical measure the quantifies the strength and direction of a linear relationship between two random variables. Use np.corrcoef().

It is also often represented as a 2-by-2 matrix like covariance.

We can interpret the result as follows.

• $corr(X, Y) \in [-1, 1]$.
• $corr(X, Y) = corr(Y, X)$.
• If it is positive correlation or $corr(X, Y) > 0$, we have as $X$ increases, $Y$’s rise on average.
  • If $corr(X, Y) = 1$, values of $X$ and $Y$ form a straight line with a positive slope.
  • If $corr(X, Y)$ is close to 1, then it is a strong relationship and values of $X$ and $Y$ lie close to a straight line.
  • If $corr(X, Y)$ is not so close to 1, then it is a weaker relationship and values of $X$ and $Y$ are distributed around but further away from a straight line.
• If it is negative covariance or $corr(X, Y) < 0$, we have as $X$ reduces, $Y$’s decrease on average.
  • If $corr(X, Y) = -1$, values of $X$ and $Y$ form a straight line with a negative slope.
  • If $corr(X, Y)$ is close to -1, then it is a strong relationship and values of $X$ and $Y$ lie close to a straight line.
  • If $corr(X, Y)$ is not so close to -1, then it is a weaker relationship and values of $X$ and $Y$ are distributed around but further away from a straight line.
• If it is zero covariance or $corr(X, Y) = 0$, then there is no linear relationship between two random variables and values scatter without a clear linear pattern.

DEMONATRATION

Continuous Random Variable

Like mathematical expectations, we cannot directly compute covariance and correlation of continuous random variables. In this case, we can convert continous random variables to discrete ones using Monte Carlo method. Softwares may have it set up in methods as follows.

# X ~ N(0, 1) 
# Y ~ N(0.5, 1.5)
data1 = np.random.normal(0, 1, 1000)
data2 = np.random.normal(0.5, 1.5, 1000)

# calculate covariance and correlation matrix
covariance_matrix = np.cov(data1, data2)
correlation_matrix = np.corrcoef(data1, data2)

:speech_balloon: FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.

Relevance: The audience of this lecture were those who have had some coding experience. Absolute beginners are encouraged to read the material and repeatedly practice coding without AI assistance to strengthen their coding and code-reading skills. Note that only necessary Python knowledge is introduced here.

Relevance: The audience of this lecture were those who could have known nothing about database or have had some experience working with data. Absolute beginners are encouraged to read and practice the basics without AI assistance to familiarize themselves with the material. Note that only necessary SQL knowledge is introduced here.

Always understand the principles first then practice.
-Dr. Thai-Ha Nguyen

In today’s post, we’ll explore Git and GitHub. They are essential tools any developer or engineer should be familiar with. We can use them for personal and collaborative purposes.