Math for AI 103

Acknowledgement: This post is based mainly on Dr. Quang-Vinh Dinh’s presentation and slides and partly on on TA, STA, and peers’ responses from the course AIO2025 offered by AI VIETNAM.

Relevance: The audience of this lecture were those who have known arithmetics or been familiar with probability and statistics. Note that only basic probability material is introduced in this Math for AI series; we’ll pick up specific concepts when in need. To complete some practice exercises, learners should know basic coding or read Python for AI 106.

In today’s post, we’ll explore basic probability including Total Probability Theorem and Bayes’ Rule. We’ll go over some AI-related practice problems and an application called simple binary classification.

---

Basic Concepts

Probability is a way to mathematicalize our real-life incidents. We’ll first look at terminolgy used to talk about those incidents formally.

Defintions

An experiment is a planned operation through which specific results are obtained. Some examples are flipping a coin and rolling a die.

An outcome is a result of an experiment. If we put all possible outcomes together in a set, we call it the sample space denoted by $\Omega$ in this post.

• Possible outcomes are displayed within braces $\{\}$.
• We can represent it in three following ways, which are demonstrated in later examples in this post,
  • list all possible outcomes
  • create a Venn diagram
  • create a tree diagram.

An event is a part/subset of the sample space, usually denoted by capital letters. It can be any combination of outcomes. There are three main types based on how likely an event is to happen.

• Certain event: an event that always occurs in an experiment, denoted by $\Omega$.
• Impossible event: an event that never occurs in an experiment, denoted by $\emptyset$.
• Random event: an even that may or may not occur in an experiment. An experiment whose outcomes are random events is called a random experiment.

We mostly work with random events and how to learn and hanlde their randomness. It’s like the law exists to guide and regulate people’s behavior, helping maintain order and fairness in society. Or analyzing stock prices is to identify the patterns, predict their future trends, and make informed investment decisions.

Let’s take a look at some examples. Assume that the coin, die, and card deck are fair and regular.

Experiment	Outcome	Sample Space	Event	Probability
Flipping a coin	{head}	{head, tail}	{tail}	$\frac{1}{2} = 0.5 = 50\%$
Rolling a die	{5}	{1, 2, 3, 4, 5, 6}	{Die is odd} = {1, 3, 5}	$\frac{3}{6} = 0.5 = 50\%$
Picking a card out of a deck	{ace of heart}	a set of 52 cards	{Card is a club} = a set of 13 clubs	$\frac{13}{52} = 0.25 = 25\%$

Practice Exercises

Exercise 1: A family has 2 children. Which of the following is a certain, random, or impossible event?

• A = "A family has 1 boy and 1 girl."
• B = "A family has 3 children."
• C = "A family has 2 children."

Exercise 2: A box contains 8 balls: 6 blue and 2 read. Pick randomly 3 balls. Which of the following is a certain, random, or impossible event?

• A = "we get 3 blue balls."
• B = "we get 3 red balls."
• C = "we get 3 balls."

Exercise 3:

Operations on Events

Consider two events A and B from the same experiment. Since events are sets, we certainly can perform set operations on them as follows. They are demonstrated by colored regions in the figures below. A is the cirle on the left and B is the circle on the right.

Notation	Operation Name	Description
$A \cap B$ or $AB$	Intersection	Both A and B happen
$A \cup B$ or $A + B$	Union	Either A or B or both happen
$A^\complement$ or $A^\prime$ or $\overline{A}$	Complement	A does not happen

The following example is solved by listing all possible outcomes in each event.

	Description	Mathematical Result
Sample Space	Rolling a die	{1, 2, 3, 4, 5, 6}
Event A	Rolling an even number	{2, 4, 6}
Event B	Rolling a number divisible by 3	{3, 6}
$AB$	Rolling a number that is even and divisible by 3	{6}
$A + B$	Rolling a number that is either even or divisible by 3	{2, 3, 4, 6}
$A^\prime$	Rolling a number that is not even	{1, 3, 5}

Practice Exercises

Exercise 1: Roll a die. Use Venn digrams to illustrate events $A, B, C, AC, AD, A + D, A^\prime, B^\prime, AF, \text{ and } F + G $.

• A = "The number rolled is odd."
• B = "The number rolled is even."
• C = "The number rolled is less than 3."
• D = "The number rolled is greater than 3."
• E = "The number rolled is greater than 4."
• F = "The number rolled is greater than or equal to 3."
• G = "The number rolled is less than 2."

Exercise 2: Write operation to describe the following diagrams. A is the cirle on the left and B is the circle on the right.

Types of Events

When dealing with probability, we often encounter two common types of events which are distinguished concepts. We’ll go over them all using events A and B above in a moment.

Mutually Exclusive Events

$A$ and $B$ are mutually exclusive events if they cannot both occur simulatenously, that is, they have no elements in common or $A \cap B = \emptyset$. Otherwise, they are non-mutally exclusive.

An example is events A and B from the practice exercise 1 above. Another example is training dataset and testing dataset in a neural network.

Independent Events

$A$ and $B$ are independent events if the occurence of A remains unaffected by that of B and vice versa, that is,

\[P(AB) = P(A) \cdot P(B).\]

$P$ represents the probability of an event which will be discussed in detail in the next section. The concept of dependent/independent events is commonly used to create a tree diagram for implication. This can help answer questions like if a treatment alleviates a disease or if there is a storm in the future with the current kind of weather. The formula derives from multiplication rule mentioned in the next section,

\[P(AB) = P(A) \cdot P(B | A).\]

Here, $P(B | A) = P(B)$ since event A happening does nothing to the occurence of event B.

If A and B are not independent, then they are dependent events.

Practice Exercises

Exercise 1: Roll a die. Let $A = \{ 4 \}$ and $B = \{ 2, 4, 6 \}$. Are A and B independent? Show the calculation.
Exercise 2: Roll a die. Based on the images below, are A and B independent? Show the calculation.

---

Probability

As mentioned, probability is a way to mathematicalize concepts, managing events’ randomness numerically. Using probability, we can talk about how likely an event occurs rather than making sets of elements.

Probability is the likehood of an event happening. That is, when we conduct a large enough number of experiments, we observe the likelihood converges to some number—it is a probability.

• $0 \leq P(A) \leq 1$ where A is an event.
• $P(\Omega) = 1$: if P(A) is close to 1, then it’s very likely that A occurs.
• $P(\emptyset) = 0$: if P(A) is close to 0, then it’s very unlikely for A to occur.

We approach different types of data in different ways in terms of probability computation. There are three primary methods to describe probability formula: classical probability, geometric probability, and experimental probability.

Classical Probability uses sets to perform a computation. We can use this method when sets are finite. Its formula is

\[P(A) = \frac{\text{number of favorable outcomes}}{\text{total number of possible outcomes}} = \frac{n(A)}{n(\Omega)}.\]

We’ll use the same event A from the above example and have

• $\Omega = \{ 1, 2, 3, 4, 5, 6 \} \Rightarrow n_{\Omega} = 6$
• $A = \{ 2, 4, 6 \} \Rightarrow n_{A} = 3$
• $P(A) = \frac{n(A)}{n(\Omega)} = \frac{3}{6} = 0.5 = 50\%$.

Geometric Probability

For situations where sets are infinite, that is, we cannot count the number of elements, we’ll use geometric probability to carry out calculations instead. It is applied in Genetic Algorithm Optimization or Large Language Models (LLMs) in Machine Learning.

\[P(A) = \frac{\text{measure of domain A}}{\text{measure of domain } \Omega}\]

where domain is the region an event can be.

While length measures the region in 1D, area measures the region in 2D. We’ll go over an example in one dimension.

Let $x$ an arbitrary real number between 0 and 1 inclusively. What is the probability of $x$ between 0 and 0.25?

• $A = (0, 0.5) \Rightarrow \text{measure of domain A} = \text{length of segment } (0, 0.25) = 0.25$
• $\Omega = (0, 1) \Rightarrow \text{measure of domain } \Omega = \text{length of segment } (0, 1) = 1$
• $P(A) = \frac{\text{measure of domain A}}{\text{measure of domain } \Omega} = \frac{0.25}{1} = \frac{1}{4}$.

Example: PI Estimation

Problem statement: Estimate PI using geometry.
Breakdown:

• $N_s$ is the number of random samples within the square generated according to uniform distribution.
• $N_c$ is the number of random samples within the circle generated according to uniform distribution.
• $s = 2 \text{ and } A_s$ are the square side and area, respectively.
• $r = 1 \text{ and } A_c$ are the circle radius and area, respectively.

Idea: Identify the relation between the number of random samples on each region and the area of such a region and utilize it to obtain their suitable ratios.

1. The circle is within the square.
   • The square sides are from -1 to 1.
   • The circle is centered at 0 with radius 1.
2. Identify the relation between the number of random samples on a region and its area.
   • The more the number of random samples is, the more the square/circle is covered.
   • The ratio of area is approximately that of the sample numbers: $\frac{N_c}{N_s} = \frac{A_c}{A_s} = \frac{s^2}{\pi r^2}$
3. Set the number of random samples on the square. Count the sample number on the circle.
   • Check if a sample is in the circle by using the circle formula.
4. Use the formula in step 1 to estimate PI.

Emperical/Expiremental Probability is the ratio of the number of event occurences to the number of attempts made. This probability method is solely based on actual data, so it’s imperative the data be accurate and not null and we carefully preprocess it.

Note the difference between this probability and the other two is that the sample space varies depending on how many times an experiment is conducted.

\[P(A) = \frac{\text{# of times occured}}{\text{# of times experiment performed}}\]

Practice Exercises

Exercise 1: Roll a die. What is the probability of getting a number greater than 3?
Exercise 2: Drawing a card from a well-shuffled deck. What is the probability of getting a king? What is the probability of getting a black card?
Exercise 3: A dart is thrown at a square dartboard such that it will land randomly on the area of the dartboard. What is the probability of it landing on the colored region?

Exercise 4: Write a program that simulates coin flips and estimates the probability of getting a head (or tail).
Exercise 5: Write a program that implements the PI estimation example.

Rules of Probability

Addition Rule refers to the probability of at least one event happening. It is known as inclusion-exclusion principle.

• Generally, we have

  \[P(A \text{ or } B) = P(A) + P(B) - P(A \text{ and } B)\] \[P(A \cup B) = P(A) + P(B) - p(A \cap B)\] \[P(A + B) = P(A) + P(B) - P(AB).\]

• If $A$ and $B$ are mutually exclusive—two events cannot occur simulatenously, then

  \[P(A + B) = P(A) + P(B)\]

  since $P(AB) = 0$ by definition.

• Since $A$ and $A^\prime$ are mutually exclusive, we can apply this rule and have the following

  \[P(A^\prime) = 1 - P(A) \text{ and } P(A) = 1 - P(A^\prime).\]

Practice Exercises

Exercise 1: Roll a die. Find the probability of getting {3, 4} using addition rule.
Exercise 2: Suppose we have the following information:

1. There is a 60% that John visits Hanoi.
2. There is a 50% that John visits HCM City.
3. There is a 30% that John visits both of the cities.

Find the probability that John visits either city.

Conditional Probability is the probability that A occurs given that B has already happened or the probability of action A based on a condition B. As such, event B happens first then event A happens. Imagine if you want to know whether there might be a rainbow tomorrow afternoon, it helps to know if it rains before then. The conditional probability is denoted and formulated as follows. We’ll discuss in detail the second equality in Bayes’ Rule.

\[P(A | B) = \frac{P(A \cap B)}{P(B)} = \frac{P(A)P(B | A)}{P(B)}\]

The first equality derives from probability definition. The condition “given B has already happened/a condition B” filters B in, saying that B is the new sample space. The condition “A occurs/action A” filters A in next, saying that from that sample space B, take only the ones from event A. The figure demonstrates this idea.

Let’s solve an example using both the idea and formula.

Statement	Definition	Formula
A fair six-sided die is rolled	$\Omega = \{1, 2, 3, 4, 5, 6\}$
A five is rolled	$A = \{3\} \Rightarrow n(A) = 1$	$P(A) = \frac{1}{6}$
An odd number is rolled	$B= \{1, 3, 5\} \Rightarrow n(B) = 3$	$P(B) = \frac{3}{6}$
Given B then A	$n(AB) = 1$	$P(AB) = \frac{1}{6}$
Probability	$P(A | B) = \frac{n(AB)}{n(B)} = \frac{1}{3}$	$P(A | B) = \frac{P(A \cap B)}{P(B)} = \frac{\frac{1}{6}}{\frac{3}{6}} = \frac{1}{3}$

Multiplication Rule
For two events, we have the following

\[P(AB) = P(A \cap B) = P(A) \cdot P(B | A) = P(B) \cdot P(A | B).\]

This formula is deduced from the conditional probability formula above,

\[P(A | B) = \frac{P(AB)}{P(B)} \text{ and } P(B | A) = \frac{P(AB)}{P(A)}\]

\begin{align} \label{conditional-multiplication} \Longrightarrow P(A) \cdot P(B | A) = P(AB) = P(B) \cdot P(A | B). \end{align}

In general, if all $A_i’s$ are true, we have the formula

\[P(A_1 A_2 \cdots A_n) = \underbrace{\underbrace{P(A_1) \cdot P(A_2 | A_1)}_{P(A_1 A_2)} \cdot P(A_3 | A_1 A_2)}_{P(A_1 A_2 A_3)} \cdots P(A_n | A_1 A_2 \cdots A_n).\]

Pro tip: If we divide a problem into separate braches and handle them in a parallel manner, use additional rule. If a problem is a sequential events, use multiplication rule.

Example

Problem statement: There are 10 identical keys, two of which can open a door. Randomly pick and try each key without replacement---discarding it if it is unusable. What is the probability that the door is opened on the third attempt?
Breakdown:

• without replacement
  • Event $A_1$ is to pick and try the first key which is right.
  • Event $A_2$ is to pick and try the second key which is right.
  • Event $A_3$ is to pick and try the third key which is right.
• $P(A_1^\prime A_2^\prime A_3)$ is the desired probability.

Idea: We pick one unit from the pool of 10 and it is wrong. Then we pick another from the rest and it is wrong. We try the third time and it is correct.

1. We pick one from 10 and it is wrong.
   • There are 2 correct keys out of 10, so $P(A_1) = \frac{2}{10}$.
   • Our chance of getting a wrong key is $P(A_1^\prime) = 1 - P(A_1) = 1 - \frac{2}{10} = \frac{8}{10}$.
2. Discarding one wrong, we pick another from the rest and it is still wrong.
   • There are 10 in total and 1 was discarded, so we have 10 - 1 = 9 keys left.
   • There are still 2 correct keys, so so $P(A_2 | A_1^\prime) = \frac{2}{9}$ .
   • Our chance of getting another incorrect given one was removed is $P(A_2^\prime | A_1^\prime) = 1 - P(A_2 | A_1^\prime) = 1 - \frac{2}{9} = \frac{7}{9}$.
3. Discard both wrong, we pick the third key from the rest and it can open the door.
   • There are 9 units left and 1 was removed, so we have 9 - 1 = 8 keys left.
   • There are 2 usable keys left and none has been picked, so we still 2 usuable keys in $A_3$.
   • Our chance of getting the first correct key given two were wrong is $P(A_3 | A_1^\prime A_2^\prime) = \frac{2}{8}$.
4. $P(A_1^\prime A_2^\prime A_3) = \frac{8}{10} \cdot \frac{7}{9} \cdot \frac{2}{8} \approx 0.156$.

Practice Exercises

Exercise 1: Find the probability of rolling an odd number given that it is a three.
Exercise 2: There are 10 identical keys, two of which can open a door. Randomly pick and try each key without replacement. What is the probability that the door is opened within at most three attempts?
Hints: At most three attempts means either one, two, or three attempts. Apply the additional rule.

Bayes’ Theorem

Total Probability Theorem

Using conditional probability, we can find the likelihood of an event happening provided that another event is true. Now what if we want to find the probability of an event H and only know that several events have already occurred and they together make up the sample space?

We can divide this problem into two stages including

• Stage 1 (choose the path you want to go with): events A_i are pairwise mutually exclusive and they combined are the sample space, which is called a complete system:
  • $A_i A_j = \emptyset$ $\forall i \neq j$
  • $\sum_{i=1}^{n} A_i = A_1 + \cdots + A_n = \Omega$
  • $P(A_1) + \cdots + P(A_n) = 1$.
• Stage 2 (take action according to the path): event H happens given that each A_i has already happened (each of them may or may not contain event H).

With this process and that all $A_i$ and $H_i = H \cap A_i$ happen separately, we’ll be able to obtain the desired probability. The idea is to divide event H into smaller events and handle each of them individually. You can think of P(H|A_i) as variables, P(A_i) as coefficients, and P(H) is the average with coefficients. The formula is as follows.

\[P(H) = P(H_1) + \cdots + P(H_n) = P(A_1) \cdot P(H | A_1) + \cdots + P(A_n) \cdot P(H | A_n)\] \[P(H) = \sum_{i=1}^{n} P(A_i) \cdot P(H | A_i)\]

Practice Exercises

Exercise 1: (Pick a Marble) There are three bags that each contain 100 marbles:

• Bag 1 has 75 red and 25 blue marbles
• Bag 2 has 60 red and 40 blue marbles
• Bag 3 has 45 red and 55 blue marbles.

Choose one of the bags at random and then pick a marble from the chosen bag also randomly. What is the probability that the chosen marble is red?
Hints: Identify two main steps and draw a tree diagram.
Exercise 2: Use columns "Studied" and "Result" of the empirical table. Compute the probability that a student studied/didn't study given their result.

Hint: Draw a tree diagram.

Bayes’ Rule

We can certainly pose another problem by reversing the situation from the Total Probability Theorem as follows.

Suppose that the probability of $H$ as a whole is known. What is the probability of $A_i$ given $H$ is true?

Take the marble example in the practice exercise 1 above. We know the probability of each color, how do we figure out which bag has more red marbles in terms of percentage? Now the problem becomes a conditional probability: What is the probability of picking a marble from bag 1/2/3 given that it is red?

Thus we can certainly use the result \eqref{conditional-multiplication} to obtain the following formula.

\[P(c | X) = \frac{P(X | c) \cdot P(c)}{P(X)}\]

where

• Posterior $P(c | X)$ is the probability of c being true given X is true
• Prior $P(c)$ is the probability of c (before carying X)
• Likelihood $P(X | c)$ is the probability of X being true given c is true
• Marginalization $P(X) \neq 0$ is the probability of X.

If $X$ is an event and $c$ is an element of a complete system ${C_1, \cdots, C_n}$ with $P(C_i) \neq 0$, then

\[P(C_i | X) = \frac{P(X | C_i) \cdot P(C_i)}{P(X)} = \frac{P(X | C_i) \cdot P(C_i)}{\sum_{i=1}^{n} P(C_j) \cdot P(X | C_j)}.\]

Example

Problem statement (Pick a Marble): There are three bags that each contain 100 marbles:

• Bag 1 has 75 red and 25 blue marbles
• Bag 2 has 60 red and 40 blue marbles
• Bag 3 has 45 red and 55 blue marbles.

Pick a random marble and it turns out to be red. What is the probability that it is from Bag 1?
Breakdown:

• $X$ is the event of picking a red marble
• $A_1$ is the event of the marble from bag 1
• $A_2$ is the event of the marble from bag 2
• $A_3$ is the event of the marble from bag 3
• $P(A_1 | X) = \frac{P(X | A_1) \cdot P(A_1)}{P(X)}$ is the probability of a marble from bag 1 given it is red.

Idea: We can use answer from the previous exercises for $P(X)$. The numerator can also be computed using the tree diagram hint.

Practice Exercises

Exercise 1: (Detect Spam Email - Simple NLP Problem) Assume that the word ‘offer’ occurs in 80% of the spam messages in my account. Also, let’s assume ‘offer’ occurs in 10% of my desired e-mails. If 30% of the received e-mails are considered as a scam, and I will receive a new message which contains ‘offer’, what is the probability that it is spam?
Hint: Draw a tree diagram.
Exercise 2: Do you notice the pattern using a tree diagram to solve this kind of problem?
Exercise 3: (Simple Classification) Use columns "Studied" and "Result" of the empirical table. Compute the probability that a student passed given their study status. Based on the probabilities, conclude if a student should study for the exam.

Hint: Draw a tree diagram.

---

FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.

Questions	Answers
What does coefficient do in total probability?	It is to evaluate the importance of each feature.
If H does not intersect with one of the Ai’s, can I exclude that Ai ?	Yes. Then, P(Ai) = 0.
Besides tree diagram, are there any techniques I can use to solve Bayes-related problem?	Yes. You can use confusion matrix for this matter.