Math for AI 103 Warm-up
Acknowledgement: This post is based mainly on TA Nguyen Quoc Thai’s presentation and slides and partly on on TA, STA, and peers’ responses from the course AIO2025 offered by AI VIETNAM.
Relevance: The audience of this lecture were those who have known arithmetics or probability and statistics.
In today’s post, we’ll explore basic probability including Total Probability Theorem and Bayes’ Rule. We’ll go over some AI-related practice problems.
Basic Terminology
Defintions
Probability is the likehood of an event happening.
- It is commonly expressed as a percentage $\%$ and ranges from $0$ to $1$ inclusively.
- In daily life, we use words like “certain”, “likely”, “maybe” or “impossible” to express it.
- $0$ is an impossible event and $1$ is a certain event.
- For example, we say, “It’s unlikely to rain in the desert” or “It’s impossible for the sun to rise in the east.” The second statement mathematically means the probability of the sun rising in the east is 0.
An experiment is a planned operation through which specific results are obtained.
- raining in the desert is an experiment
- flipping a coin is an experiment.
An outcome is a result of an experiment. The sample space of an experiment is the set of all possible outcomes.
- Possible outcomes are displayed within braces $\{\}$.
- We can represent it in three following ways, which are demonstrated in later examples in this post,
- list all possible outcomes
- create a Venn diagram
- create a tree diagram.
An event is a part/subset of the sample space. It can be any combination of outcomes.
Let’s take a look at some examples. Assume that the coin, die, and card deck are fair and regular.
| Experiment | Sample Space | Event | Probability |
|---|---|---|---|
| Flipping a coin | {head, tail} | {tail} | $\frac{1}{2} = 0.5 = 50\%$ |
| Rolling a die | {1, 2, 3, 4, 5, 6} | {Die is even} = {2, 4, 6} | $\frac{3}{6} = 0.5 = 50\%$ |
| Picking a card out of a deck | a set of 52 cards | {Card is a club} = a set of 13 clubs | $\frac{13}{52} = 0.25 = 25\%$ |
Relations of Events
Consider two events A and B from the same experiment.
Implication
- Event A implies event B means if event A occurs, then event B occurs as well.
- Mathematically, $A \Rightarrow B$ means $A \subseteq B$.
Equivalent
- Event A equals event B means A implies B and B implies A.
- Mathematically, $A \Rightarrow B$ and $B \Rightarrow A$ means $A \Leftrightarrow B$.
Operations on Events
Again, consider two events A and B from the same experiment. Some primary operations on them as follows.
| Notation | Operation Name | Description |
|---|---|---|
| $A \cap B$ or $AB$ | Intersection | Both A and B happen |
| $A \cup B$ or $A + B$ | Union | Either A or B or both happen |
| $A^\complement$ or $A^\prime$ or $\overline{A}$ | Complement | A does not happen |
The following example is solved by listing all possible outcomes in each event.
| Description | Mathematical Result | |
|---|---|---|
| Sample Space | Rolling a die | {1, 2, 3, 4, 5, 6} |
| Event A | Rolling an even number | {2, 4, 6} |
| Event B | Rolling a number divisible by 3 | {3, 6} |
| $AB$ | Rolling a number that is even and divisible by 3 | {6} |
| $A + B$ | Rolling a number that is either even or divisible by 3 | {2, 3, 4, 6} |
| $A^\prime$ | Rolling a number that is not even | {1, 3, 5} |
Practice Exercises
Exercise 2: Roll a die. If A is an event of getting even sides and B is an event of getting a number greater than or equal to 4. What is $AB$?
Exercise 3: Roll a die. If A is an event of getting a number greater than 2 and B is an event of getting a number greater than or equal to 4. What is $A + B$?
Probability
There are two ways to describe probability formula: classical probability and geometric probability.
Classical Probability is represented by the formula
\[P(A) = \frac{\text{number of favorable outcomes}}{\text{total number of possible outcomes}} = \frac{n(A)}{n(\Omega)}.\]We’ll use the same event A from the above experiment and have
- $\Omega = \{ 1, 2, 3, 4, 5, 6 \} \Rightarrow n_{\Omega} = 6$
- $A = \{ 2, 4, 6 \} \Rightarrow n_{A} = 3$
- $P(A) = \frac{n(A)}{n(\Omega)} = \frac{3}{6} = 0.5 = 50\%$.
Geometric Probability
\[P(A) = \frac{\text{measure of domain A}}{\text{measure of domain } \Omega}\]where domain is the region an event can be.
While length measures the region in 1D, area measures the region in 2D. We’ll go over an example in one dimension.
Let $x$ an arbitrary real number between 0 and 3 inclusively. What is the probability of $x$ closer to 0 than 1?
- Breakdown: Only numbers less than 0.5 are closer to 0 than 1. That is the region where $x$ can be to satisfy the condition.
- $A = (0, 0.5) \Rightarrow \text{measure of domain A} = \text{length of segment } (0, 0.5) = 0.5$
- $\Omega = (0, 3) \Rightarrow \text{measure of domain } \Omega = \text{length of segment } (0, 3) = 3$
- $P(A) = \frac{\text{measure of domain A}}{\text{measure of domain } \Omega} = \frac{0.5}{3} = \frac{1}{6}$.
Practice Exercises
Exercise 2: There are 3 red marbles and 2 green marbles. What is the probability of getting a red one?
Exercise 3: Assume $x$ selected from the set ${-2, -1, 0, 1, 2}$. What is the probability of $x^2 > 0$?
Exercise 4: Roll 2 dice. What is the probability of getting two sides summed up to 5?
Exercise 5: Using the 1D geometric probability example, what is the probability if x is closer to $2.5$ than $1$?
Exercise 6: A dart is thrown at a circular dartboard such that it will land randomly on the area of the dartboard. What is the probability of it landing closer to the center "Success" than to the edge?
Rules of Probability
Addition Rule refers to the probability of at least one event happening.
-
Generally, we have
\[P(A \text{ or } B) = P(A) + P(B) - P(A \text{ and } B)\] \[P(A \cup B) = P(A) + P(B) - p(A \cap B)\] \[P(A + B) = P(A) + P(B) - P(AB).\] -
If $A$ and $B$ are mutually exclusive—two events cannot occur simulatenously, then
\[P(A + B) = P(A) + P(B)\]since $P(AB) = 0$ by definition.
-
Since $A$ and $A^\prime$ are mutually exclusive, we can apply this rule and have the following
\[P(A^\prime) = 1 - P(A) \text{ and } P(A) = 1 - P(A^\prime).\]
Practice Exercises
Exercise 2: Find the probability of getting a number different from 1 and 6 when rolling a fair six-sided die.
Conditional Probability is the probability that A occurs given that B has already happened. Imagine if you want to know whether there might be a rainbow tomorrow afternoon, it helps to know if it rains before then. The conditional probability is denoted and formulated as follows. We’ll discuss in detail the second equality in Bayes’ Rule.
\[P(A | B) = \frac{P(A \cap B)}{P(B)} = \frac{P(A)P(B | A)}{P(B)}\]The first equality derives from probability definition. The condition “given B has already happened” says that all the possible outcomes are from event B. The condition “A occurs” says that from those outcomes of event B, take only the ones from event A. The figure demonstrates this idea.
Let’s solve an example using both the idea and formula.
| Statement | Definition | Formula |
|---|---|---|
| A fair six-sided die is rolled | $\Omega = \{1, 2, 3, 4, 5, 6\}$ | |
| A five is rolled | $A = \{5\} \Rightarrow n(A) = 1$ | $P(A) = \frac{1}{6}$ |
| An odd number is rolled | $B= \{1, 3, 5\} \Rightarrow n(B) = 3$ | $P(B) = \frac{3}{6}$ |
| Given B then A | $n(AB) = 1$ | $P(AB) = \frac{1}{6}$ |
| Probability | $P(A | B) = \frac{n(AB)}{n(B)} = \frac{1}{3}$ | $P(A | B) = \frac{P(A \cap B)}{P(B)} = \frac{\frac{1}{6}}{\frac{3}{6}} = \frac{1}{3}$ |
Multiplication Rule
From the conditional probability formula, we can deduce a formula for two events. The formula for a general case is provided below and will be explained in Math for AI 103.
Let’s first see how conditional probability formula implies the first formula here and then take an example to see how these formulae work.
\[P(A \\| B) = \frac{P(AB)}{P(B)} \text{ and } P(B \\| A) = \frac{P(AB)}{P(A)}\]\begin{align} \label{conditional-multiplication} \Longrightarrow P(A) \cdot P(B | A) = P(AB) = P(B) \cdot P(A | B) \end{align}
Example
Breakdown:
- at once means together in one time
- without replacement means one after another and not returning to the original pool.
- Event $A_1$ is to pick the first non-defective unit.
- Event $A_2$ is to pick the second non-defective unit.
- Event $A_3$ is to pick the last non-defective unit.
- Both approaches give the same result because three distinct items are chosen from the same fixed population and the order of selection does not matter.
- We pick one from 100 and hope it is non-defective.
- There are five defectives out of 100, so there are 100 - 5 = 95 non-defectives in $A_1$.
- Our chance of getting a non-defective is $P(A_1) = \frac{95}{100}$.
- Given that the first selected product was good, we pick another from the rest and hope it is non-defective.
- There are 100 in total and 1 was picked, so we have 100 - 1 = 99 units left.
- There are 95 non-defectives in total and 1 was assumed to be picked, so we have 95 - 1 = 94 non-defective units left in $A_2$.
- Our chance of getting another non-defective given one has already been picked is $P(A_2 | A_1) = \frac{94}{99}$.
- Given that both selected items were fine, we pick the last unit from the rest and hope it is non-defective.
- There are 99 units left and 1 was picked, so we have 99 - 1 = 98 units left.
- There are 94 non-defectives left and 1 was assumed to be picked, so we have 94 - 1 = 93 non-defective units in $A_3$ now.
- Our chance of getting another non-defective is $P(A_3 | A_1 A_2) = \frac{93}{98}$.
- $P(A_1 A_2 A_3) = \frac{95}{100} \cdot \frac{94}{99} \cdot \frac{93}{98} \approx 0.856$.
Practice Exercises
Exercise 2: Suppose you pick a card from a standard deck. Find the probability of getting a J, Q, and K if we know that card is red.
Exercise 2: Suppose you take out two cards from a standard deck of cards one after another without replacement. What is the probability that the first is the ace of spade and the second if a heart?
Total Probability Theorem
Using conditional probability, we can find the likelihood of an event happening provided that another event is true. Now what if we want to find the probability of an event H and only know that several events have already occurred?
We can divide this problem into two stages including
- Stage 1: events $A_i$ are pairwise mutually exclusive and they combined are the sample space, which is called a complete system:
- $A_i A_j = \emptyset$ $\forall i \neq j$
- $\sum_{i=1}^{n} A_i = A_1 + \cdots + A_n = \Omega$
- $P(A_1) + \cdots + P(A_n) = 1$.
- Stage 2: event $H$ happens given that each $A_i$ has already happened (each of them may or may not contain event H).
With this process and that $A_i$ and $H_i = H \cap A_i$ happen separately, we’ll be able to obtain the desired probability. The idea is to divide event $H$ into smaller events and handle each of them. The formula for this idea is as follows.
\[P(H) = P(H_1) + \cdots + P(H_n) = P(A_1) \cdot P(H | A_1) + \cdots + P(A_n) \cdot P(H | A_n)\] \[P(H) = \sum_{i=1}^{n} P(A_i) \cdot P(H | A_i)\]
Practice Exercises
Hints: Complete the tree diagram below and apply the formula.
- Bag 1 has 75 red and 25 blue marbles
- Bag 2 has 60 red and 40 blue marbles
- Bag 3 has 45 red and 55 blue marbles.
Hint: Draw a tree diagram as above.
Bayes’ Rule
We can certainly pose another problem by reversing the situation from the Total Probability Theorem as follows.
Suppose that the probability of $H$ as a whole is known. What is the probability of $A_i$ given $H$ is true?
Take the car shop example. We know the probability of defective units, how do we figure out which company provides more defective products in terms of percentage? Now the problem becomes a conditional probability: What is the probability of a product coming from each company given that the product is defective?
Thus we can certainly use the result \eqref{conditional-multiplication} to obtain the following formula.
\[P(c | X) = \frac{P(X | c) \cdot P(c)}{P(X)}\]where
- Posterior $P(c | X)$ is the probability of c being true given X is true
- Prior $P(c)$ is the probability of c
- Likelihood $P(X | c)$ is the probability of X being true given c is true
- Marginalization $P(X)$ is the probability of X.
If c is an element of a complete system ${A_1, \cdots, A_n}$ and X is an event $H$ with $P(A_i) \neq 0$, then
\[P(A_i | H) = \frac{P(H | A_i) \cdot P(A_i)}{P(H)} = \frac{P(H | A_i) \cdot P(A_i)}{\sum_{i=1}^{n} P(A_j) \cdot P(H | A_j)}.\]Example
Breakdown:
- $H$ is the event of a unit being defective
- $M$ is the event of a unit from company M
- $N$ is the event of a unit from company N
- $P(M | H) = \frac{P(H | M) \cdot P(M)}{P(H)}$ is the probability of a unit from M given it is defective.
Practice Exercises
Exercise 2: (Pick a Marble) Use the marble example from previous exercises. Pick a random marble and it turns out to be red. What is the probability that it is from Bag 1?
Exercise 3: (Detect Spam Email - Simple NLP Problem) Assume that the word ‘offer’ occurs in 80% of the spam messages in my account. Also, let’s assume ‘offer’ occurs in 10% of my desired e-mails. If 30% of the received e-mails are considered as a scam, and I will receive a new message which contains ‘offer’, what is the probability that it is spam?
Hint: Draw a tree diagram.
Exercise 4: Do you notice the pattern using a tree diagram to solve this kind of problem?
Exercise 5: A card is lost from a pack of 52 cards. From the remaining cards two are drawn randomly and found to be both clubs. Find the probability that the lost card is also a clubs.
Hints: Draw a tree diagram. The given condition is that two cards are drawn at random.
Exercise 6: Solve exercise 5 in a one-line argument.
Hints: Use only logic and definition of probability.
FAQ
Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.
| Questions | Answers |
|---|---|
Enjoy Reading This Article?
Here are some more articles you might like to read next: