Machine Learning 101

Acknowledgement: This post is based mainly on Dr. Quang-Vinh Dinh’s presentation and slides and partly on on TA, STA, and peers’ responses from the course AIO2025 offered by AI VIETNAM.

Relevance: The audience of this lecture were those who have read Machine Learning 101 Warm-up, had some coding experience, and known a bit of AI knowledge.

In today’s post, we’ll explore K-Means and Unsupervised Learning. We’ll also discuss some applications in AI including tabular, image, and text data.

A dataset in Supervised Learning (SL) comes with two parts: X (the features) and Y (the labels). SL is used to solve classification and regression problems where we need the input data X and their results Y in order for a model to learn. In Unsupervised Learning (USL), a dataset only has X. Since there is no Y, USL cannot solve problems like classification or regression. Instead it can work on clustering problems which aim to group data points based on some property/condition; different properties/features can give different output.

Motivation

Let’s discuss a real-life problem. We have a dataset of sales from customers; this dataset does not give any information on customers, that is, is not labeled. The data is scattered and the whole dataset is not compactly represented.

The questions become, “How can we extract some useful information? More specifically, how can we classify customers into VIP and normal groups?”.

This is a typical example for the clustering problem.

For this particular question, we can think of the way and amount each group spends. The potential VIP tends to spend much more than a normal customer. Qualitatively, this can result in a clear divider between two groups. Visually, the customers are grouped into two sets which do not intersect/overlap with each other.

Nonetheless, to conclude it quantitatively, we need to have numbers to support our conjecture. Let’s experiment with a sample dataset.

Note: Even though the solution presented in the Sample works for a toy dataset, some problems needs addressing with this approach for it to work in real-world applications.

• Manually grouping and trying many possible of them for an actual dataset are very inefficient.
  • There is no clear way on how to make a grouping better.
• Unsystematically forming/guessing a grouping leads to a center biased by the grouping.
  • It’s hardly possible to get a proper center and its cluster.

To uplift the solution we discussed above, we can reverse the process by initially picking centers and then forming clusters.

$K$-Means

The improved version of the example above follows the exact procedure for grouping $n$ observations into $K$ clusters based on the distance in $K$-Means.

Procedure

We should always preprocess the data first. Then we can apply $K$-Means to a ML model in the following steps.

1. Initialize the value of $K$.
2. Select random $K$ centroid.
3. Assign each data point to the closest centroid.
4. Update a new centroid of each cluster.
5. If not convergent, go to step 3. Otherwise, stop.

flowchart LR
    A((<strong>Data processing</strong><br> select K</strong>)) ---> B((<strong>Select random<br> K centroids</strong>)) ---> C((<strong>Assign each data point<br> to the closest centroid</strong>)) ---> D((<strong>Update centroids</strong>)) --->|Not Convergent| E((<strong>If not convergent,<br> go to step 3</strong>))
    D --->|Convergent| F((<strong>Stop</strong>))

Note: $K$-Means always converges to a local minimum but does not guarantee convergence to global minimum (optimum). The result is dependent on the initialization.

$K$-Means Algorithm

You may question, “Why do we need to cluster data points? What do we do with them?

This question is on point. In USL, data is unlabeled, thus clustering data points into appropriate and suitable groups is the key to label the entire dataset based on the problem requirement.

%%{init: {'flowchart': {'curve': 'basis'}} }%%
flowchart TB
    %% Main Flow
    subgraph Algorithm ["Algorithm/Procedure"]
        direction TB

        subgraph IterateCentroids[" "]
            direction TB

            subgraph UpdateCentroid["Step 3: Update Centroids"]
                direction TB
                Recompute["Based on data points\n in the same centroid,\n re-compute the centroids"]:::dashedgreen
            end

            subgraph ClusterStep["Step 1: Find K Clusters"]
                direction BT
                RandomPickup["Random pickup"]:::dashed
                KCentroids1["K-Centroids"]:::core
                RandomPickup --> KCentroids1
            end
        end

        subgraph LabelStep["Step 2: Labeling data points"]
            direction LR
            AssignLabels["Assign the label of the nearest\n centroid for all the data points"]:::dashedorange
            NLabelled["N-data points (labelled)"]:::coreorange
            AssignLabels --> NLabelled
        end
    end

    subgraph Idea [Main Idea]
        direction TB
        Inputs["Inputs:\n• Unlabelled dataset\n• Number of Clusters (K)"]:::input
        Outputs["Outputs:\n• Centroids\n• Dataset labelled"]:::output
        Inputs ------> Outputs
    end

    NLabelled -.->|"Update centroids"| UpdateCentroid
    UpdateCentroid --> KCentroids1

    %% Cross connection
    KCentroids1 -.->|"Update labels"| AssignLabels

    %% Styling Definitions
    classDef input fill:#e6eef9,stroke:#7293ba,stroke-width:2px,font-family:serif;
    classDef output fill:#e6faeb,stroke:#628e6e,stroke-width:2px,font-family:serif;
    classDef dashed stroke:#7293ba,stroke-width:1px,stroke-dasharray: 6 4,fill:#f5faff,font-family:serif;
    classDef core fill:#c4e0fa,stroke:#3b6fa6,stroke-width:2px,font-weight:bold,font-family:serif;
    classDef dashedgreen fill:#f0faf4,stroke:#628e6e,stroke-width:2px,stroke-dasharray: 6 4,font-family:serif;
    classDef dashedorange fill:#faf2ea,stroke:#cd7a39,stroke-width:2px,stroke-dasharray: 6 4,font-family:serif;
    classDef coreorange fill:#fad1ae,stroke:#cd7a39,stroke-width:2px,font-weight:bold,font-family:serif;
    classDef labelHeader fill:#c4e0fa,stroke:#3b6fa6,stroke-width:2.5px,font-weight:bold,text-align:center;
    classDef orangeHeader fill:#ed8f33,stroke:#cd7a39,stroke-width:2.5px,font-weight:bold,color:#fff,text-align:center;
    class ClusterStep labelHeader
    class UpdateCentroid labelHeader
    class LabelStep orangeHeader

Formula

We can write these steps in mathematical expressions.

The goal of the clustering problem is to make the groups tight and compact while still covering all the data. Mathematically, it can be expressed as follows.

Tabular Data

Practice Exercises

Exercise 1: Use the following Iris data table and step-by-step $K$-Means procedure with Euclidean distance to label data points. Fill in the table below for each iteraton. Take $K = 2$, $c_0 = 1.4$, and $c_1 = 1$. Check the answer with code snippet below.

Index	Length	Distance to $c_0$	Distance to $c_1$	Label	New Centroids
0	1.4
1	1
2	1.5
3	3.1
4	3.8
5	4.1

Exercise 2: Do the same as exercise 1 and make suitable tables for the age-expenditure table with

1. $K = 2$, $c_0 = (18, 80)$ and $c_1 = (22, 85)$$
2. $K = 3$, $c_0 = (22, 85)$, $c_1 = (30, 50)$, and $c_2 = (34, 64)$

Exercise 3: Based on the code snippet for exercise 1 below and the following exercises, write code for exercise 2.
Exercise 4: What is the output of the following? Explain your answer.

data = np.random.rand(64,2)
centroids = np.random.rand(5,2)
distances = (data - centroids)**2

print(data.shape)
print(centroids.shape)
print(distances.shape)

data = np.random.rand(64,2)

data1 = data[:, np.newaxis, :]
data2 = data.reshape(1, 1, -1)

print(data1.shape)
print(data2.shape)

data = np.random.rand(34,4)
centroids = np.random.rand(3,4)
distances = np.sqrt((data[:, np.newaxis, :] - centroids)**2)

print(data.shape)
print(centroids.shape)
print(distances.shape)

Exercise 5: Which one gives the solution?

# 32 point, 3 dimensions (3 features), 4 centroids
squared_norm = np.random.rand(32,3,4)
# what is the distance between points and centroids?
np.sqrt(squared_norm.sum(axis=0))
np.sqrt(squared_norm.sum(axis=1))
np.sqrt(squared_norm.sum(axis=2))

# 32 point, 3 dimensions (3 features), 2 centroids
distances_detail = np.random.rand(32,3,2)
# what is the distance between points and centroids?
np.linalg.norm(distances_detail,axis=0)
np.linalg.norm(distances_detail,axis=1)
np.linalg.norm(distances_detail,axis=2)

Below is the code snippet for the practice exercise 1 just above.

import numpy as np

X = np.array([1.4, 1.0, 1.5, 3.1, 3.8, 4.1])

# step 1
k = 2

# step 2
centroids = X[:k]

X = X.reshape(6,1)

while True:
    # step 3a
    distances = np.sqrt((X - centroids)**2)
    print(f'distances:\n {distances}')

    # step 3b
    labels = np.argmin(D, axis=1)
    print(f'labels:\n {labels}')

    # step 4: update centroids
    new_centroids = np.array([X[L == i].mean() for i in range(k)])

    # step 5: check convergence
    if np.all(centroids == new_centroids):
        break

    # step 2: centroids
    centroids = new_centroids
    print(f'centroids:\n {centroids}')

We can also use an available package to solve the problem as well.

from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=2, max_iter=10)
kmeans.fit(X.reshape(6,1))
labels = kmeans.labels_
print(labels)
print(kmeans.inertia_)

for x, label in zip(X, labels):
    print(f'Cluster {label}: {x}')

In this example, our dataset consists of six data points $\{x_1, \cdots, x_6\}$ which are all one-dimensional real numbers, denoted $\mathbb{R^1}$ or $\mathbb{R}$. The initial $K = 2$ determines two clusters $\{S_0, S_1\}$.

Formula

Another critical obstacle when working with $K$-Means algorithm is to obtain the optimal $K$ or the optimal number of clusters. To do so, we need to determine how well the data points spread out from the centroids of their clusters.

The Within-Cluster Sum of Squares (WCSS) works like a signal for the best $K$ (like in the Sample example); it measures how compact the clusters are using a clustering procedure, making it easier to compare different $K$-clustering results.

The value of WCSS declines to 0 as $K$ grows. Using WCSS, we can figure out the optimal number of clusters via a method called the Elbow Method. This method aims to pick the elbow value where the rate of decrease for WCSS slows down sharply.

Image and Text Data

We can apply $K$-Means to image and text datasets as well. Before doing so, we must preprocess the data so that it becomes pertain to the algorithm.

Image Data

An image dataset can have many images. An image can be described as a matrix whose shape depends on its image type.

To treat this kind of dataset as input of $K$-Means algorithm, we first need to make each image into a vector just like a row/data point in the tabular data above. That is, we flatten matrices.

In practice, Deep Learning uses a magic function to process an image to extract its important parts which determine the content of the image. Just like our human brain, when we look at a cat or a dog image, we can tell what kind they are based on their face. The model completing this process is called a pretrained model. This reduces the size of data by a significant amount.

Text Data

Similar to image data, there is a magic function to learn important words which speak the main content of an article.

For text, we need to convert it into numeric values using vectorization algorithms before using $K$-Means.

FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.