Math for AI 104

Acknowledgement: This post is based mainly on Dr. Quang-Vinh Dinh’s presentation and slides and partly on on TA, STA, and peers’ responses from the course AIO2025 offered by AI VIETNAM.

Relevance: The audience of this lecture were those who have read Math for AI 103 or been familiar with probability and statistics. Note that only basic Probability material is introduced here; we’ll pick up specific concepts when in need.

Our world operates based on creativity, so we should always be ready to be creative. If we are given a task and it cannot be completed using available tools. It is a good signal that we can be creative to make great things happen.
-Dr. Quang-Vinh Dinh

In today’s post, we’ll explore Naive Bayes Classifier and Gaussian Function. We’ll also discuss some applications in AI including classification problems.

Classification Problem Overview

A classification problem in Machine Learning involves training a model to learn patterns and predict outcomes based on specific input data. Outcomes or targets, often listed in a classifying column, is called a class, category, label, or ground truth. The term ground truth is commonly used for image datasets. Input data, listed in other columns, is named a feature.

In the table above, there are three features including “Confident”, “Studied”, and “Sick”. Using them, we want to a model to provide some value of the class “Result”.

Review

Recall $A$ and $B$ are independent events if the occurence of A remains unaffected by that of B and vice versa, that is, $P(AB) = P(A) \cdot P(B) \text{ or } P(A | B) = P(A) \text{ or } P(B | A) = P(B)$. This concept significantly simplifies the process of working with large datasets since we only need to know the probability of events separately.

Events $A$ and $B$ are conditionally independent given $C$ if and only if

Recall for a complete system $A_1, \cdots, A_n$ and an event $H$, Total Probability Theorem says

The likelihood of A_i and that of H given A_i is true are included since the occurence of event H only makes sense when event A_i has happened. In classification problems, features only become meaningful to a training dataset within the context of the class. Without classes, models have no reference point to interpret what the features represent.

Bayes’ Rule says if $X$ is an event and $c$ is an element of a complete system ${C_1, \cdots, C_n}$ with $P(C_i) \neq 0$, then

Note: The fact that such unrealistic assumptions are made gives right to the word naive in the Naive Bayes algorithm, which derives $P(AB) = P(A) \cdot P(B)$. For example, when we use this formula to classify emails with the word “Free Money”, it says P(Free Money) = P(Money Free) or “Free” is unrelated to “Money”, which is often not true. See practice exercises for detailed computations.

Practice Exercises

Exercise 1: Using the following table, analyze the relationship between "Studied" and "Result" in each situation?

1. P(Studied = No, Result = Fail)
2. P(Studied = Yes, Result = Fail)
3. P(Studied = No, Result = Pass)
4. P(Studied = Yes, Result = Pass)

Exercise 2: Using the above table assume that the relationships between "Studied" and "Result" and "Sick" and "Result" are both independent, analyze the probability of "Studied" and "Sick" given "Result" in each situation?

1. P(Studied = No, Sick = Yes | Result = Pass)
2. P(Studied = Yes, Sick = No | Result = Fail)
3. P(Studied = Yes, Sick = Yes | Result = Fail)

Exercise 3: Out of 6 students, there are 3 passes and 3 fails. Out of passes, there is one student who didn't study. Out of fails, there are two. What is the probability of a student who didn't study?
Exercise 4: Can two dependent events be conditionally independent? Explain why or why not. Provide an example.
Exercise 5: Can two independent events be conditionally dependent? Explain why or why not. Provide a proof if possible. Provide an example.

Bernoulli Naive Bayes Classifier (BNBC) for Discrete Random Variable (RV)

Bernoulli Naive Bayes is a machine learning algorithm tailored for discrete and binary-valued feature problems. Binary-valued features can be Yes/No or True/False. It uses conditional probabilities to make predictions, whether we’re working with one feature or many at once.

One feature

We talked about this in the previous post. Try practice exercises to familiarize yourself with it.

Practice Exercises

Exercise 1: Use feature "Studied" and "Result" of the empirical table. Compute the probability that a student studied/didn't student given their result.

Exercise 2: Solve exercise 1 without the formula.
Hints: What is $\sum_{i} P(Studied | Pass)$?

Multiple Features

For multiple-featured problems, apply Bayes’ Rule and conditional independence assumptions.

Let $C$ be a class and $X_1, \cdots, X_n$ be features (also known as random variables introduced in Math for AI 105 Warm-up). The formula for the probability of an event given a few other events has already occurred.

where $C_k$’s are all labels.

Practice Exercises

The naive assumptions are applied in both exercises.
Exercise 1: Let R, Co, St, and Si be random variables short for "Result", "Confident", "Studied", and "Sick". Compute $P(R | Co, St, Si)$ where R = Pass, Co = Yes, St = Yes, Si = Yes.

Exercise 2: Ignore the last row, what is the relationship among random variables given the target column is "Confident"?

Gaussian Function

All we’ve been through so far involves events where we can count the number of features. You may wonder, what if given datasets contain continuous values which lie within a given range instead of a finite set?

Definition

A distribution is a collection of data shared among a range of values on a variable.

The probability distribution for a random variable describes how the occurence of possible outcomes for an experiment are distributed over that variable. Its graph is bell-shaped, maybe skewed or not, as in figure. The Gaussian function is its special type known as normal distribution where its graph is symmetric bell-shaped curve as in figures.

where

• $-\infty < x < \infty$
• $\mu_{X} = \frac{1}{n} \sum_{i=1}^{n} x_i$ is mean of the dataset $X$
• $\sigma^2 = var(X) = \frac{1}{n} \sum_{i=1}^{n} (x_i - \mu)^2$ is the variance of the dataset $X$.

It’s worth pointing out a few fascinating properties of Gaussian distribution graph.

• The graphs are symmetric bell-shaped curve.
• Changing $\mu$ or $\sigma^2$ does not change the bell shape of the normal distribution—it either shifts or stretches the standard graph (Figure ).
• Any Gaussuan distribution can be converted to a standard normal distribution.
• The area under the curve of the standard graph is $1$.

The third characteristic implies that we can use its area function as probability for continuous random variables with area being always non-negative and uninterrupted. For example, for the Iris flower case, the length and width can be any real number within some range. Thus using a function that can handle continuity like Gaussian would solve the problem.

Applications in AI
A Gaussian filter uses the Gaussian function to obtain a weighted avergae of surrounding pixels in Image Smoothing. A bilateral filter uses this concept with information of pixel color to perform better on edges of objects.

Central Limit Theorem is the idea of using a sample—a subset of the entire population—for analysis. You can take any example, but here we’ll use an image whose population contains pixel colors of the entire image. First, randomly select 30 pixels and compute their color mean. We repeat this process 10000 times. Eventually we can create a histogram of these averages. Even if the original values of the entire population are not normally distributed, the sample’s mean values are inclined to be.

Gaussian Naive Bayes Classifier (GNBC) for Continuous RV

Gaussian Naive Bayes is a machine learning algorithm that combines the Gaussian function and Bayes’ Theorem to solve continuous feature problems.

One Feature

The probability at one value $P(X | C) = 0$ at all times in the case of continuous random variables. So, if we use a real probability function, it’s impossible to make sense of a dataset.

We want a function that can capture the idea of likelihood. That function is called probability density function, which measures the density around a value. The higher the density is, the more likely the value is relative to others. As Gaussian area function acts as probability, the Gaussian function itself becomes such a density function.

where $\mu$ and $\sigma^2$ are the mean and variance of class $C$.

Note: The density function can be evaluated greater than 1 when the graph narrows down and stretches vertically as long as the total area under the curve is still 1. This ensures that all the probabilities are from 0 to 1, which is consistent with the foundational rule of probability theory. Essentially, PDF does not measure a probability.

Practice Exercises

Exercise 1: Given the following, compute $P(Length = 3.0)$.

• $P(Cat = 0) = 0.5$
• $P(Cat = 1) = 0.5$
• $f(Length = 3.0 | Cat = 0) = 0.01$
• $f(Length = 3.0 | Cat = 1) = 0.01$

Exercise 2: Use the following table to compute the probability of each category given the length is 3.0 units.

Exercise 3: What is the sum of the probabilities in exercise 2. Explain why it is that value.
Exercise 4: Write a program to compute the probability in exercise 2.
Hints: Use two ways: using algorithm and using packages
Exercise 5: Can "Length" be the category? Explain why or why not.

Multiple Features

As we’ve seen the Iris flower table has two features. This means we can use the probability of each category given both of them.

Practice Exercises

Exercise 1: How many Gaussian functions do we need to compute the probability of each category given two features?

Exercise 2: Use the above table to compute the probability of each category given the length is 4.1 units and the width is 2.9 units.
Exercise 3: Write a program to compute the probability in exercise 2.
Hints: Use two ways: using algorithm and using packages.
Exercise 4: What is the sum of the probabilities in exercise 2? Explain why it is that value.
Exercise 5: Use the following table and compute the probability of each category given both features.

Multiple Features - Multiple Classes

We can definitely apply the same idea to compute the probability where there are multiple features and multiple classes.

Practice Exercises

Use the following table to compute the probability of each category given the length is 5.2 units and the width is 2.4 units.

FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.