Math for AI 105 Warm-up

Acknowledgement: This post is based mainly on TA. Vo Huynh Quang Nguyen’s presentation and slides and partly on on TA, STA, and peers’ responses from the course AIO2025 offered by AI VIETNAM.

Relevance: The audience of this lecture were those who have read Math for AI 104 or been familiar with probability and statistics. Note that only basic probability material is introduced here; we’ll pick up specific concepts when in need.

In today’s post, we’ll explore basic probability concepts: random variables, probability distribution, and descriptive statistics.

Random Variable (RV)

Definition

Let’s use some beginner-friendly scenarios to introduce the idea of a random variable.

From above cases, we can think of a random variable as a real number depending on the outcome of a random experiment.

Formally, a random variable $X$ is a function $X: \Omega \to \mathbb{R}$, $s \mapsto X(s)$ where $\Omega$ is the sample space—set of outcomes and $\mathbb{R}$ is the set of real numbers. Let’s delve into two types of random variables and their examples to nail this concept.

Discrete RV - Probability

A discrete random variable $X$ is a function $X: \Omega \to \mathbb{R}$ where $\Omega$ is a finite or countably infinite set. Since the range of this function contains numbers representing mutually exclusive events, we can measure probability on random variables and the sum of all those must be 1.

Probability on random variables are known as probability distribution. Probability mass and density are common probability distributions.

Probability Mass Function (PMF), denoted by $p(x)$ or $p_X(x)$, measures the probabiltiy of possible values of a discrete random variable, that is, $p(x) = p_X(x) = P(X = x)$.

Cummulative Distribution Function (CDF), denoted by $F(x)$ or $F_X(x)$, is the cumulative probability of of all possible values less than equal to a specific number, that is, $F_X(x) = P(X \le x)$.

We’ll see examples demonstrated in two following tables.

Below, consider the number of heads in the sequence as a random variable.

Random Variables	PMF	CDF
$X(s) = 2 \Rightarrow P(X = 2) = \frac{3}{8}$	$p(2) = \frac{3}{8}$
$1 < X(s) \le 3 \Rightarrow P(1 < X \le 3) = \frac{4}{8}$
$X(s) \le 2 \Rightarrow X \in \\\{0, 1, 2\\\} \Rightarrow P(X \le 2) = \frac{7}{8}$		$F(2) = \frac{7}{8}$

Continuous RV - Probability

A continuous random variable $X$ is a function $X: \Omega \to \mathbb{R}$ where $\Omega$ is an uncountabe set. The normal distribution mentioned in Math for AI 104 is one of the most well-known probability models for continuous random variables .

Let’s see a couple of examples.

1. Imagine we’re looking at the second hand of a clock and take a random snapshot at an unknown moment during a one-minute period.
   • Define $X(0, 12]$ = angle $\theta$ (in degrees) between the second hand and the 12 o’clock position.
   • The second hand sweeps smoothly in a full circle ($360^\circ$) every minute.
   • So $X \in (0, 360]$.
2. Adult female/male heights.
   • A person’s height can be any positive real number.
   • So $X \in (0, \infty)$.

PMF for continuous random variables is always 0 since the sample space is so large that it makes the occurence of one value become insignificant. On the other hand, PDF for continuous random variables can be computed using the Gaussian function.

Mathematical Expectations

Below are some concepts used for discrete random variables. We’ll briefly go over mathematical expectations for continous random variables right after.

Expected Value

Expected value of a discrete random variable X, denoted by $E[X]$, represents the weighted average value of X and computed as follows. It is often referred to as the mean of probability distribution but tremendously more useful when the size of the sample space is unknown.

Some important properties of expected values are

• $E[\alpha X] = \alpha E[X]$
• $E[\alpha X + b] = \alpha E[X] + b$.

Variance

Variance of a random variable X, denoted as $var(X)$, is the weighted average of squared distance from the mean. The reason behind squared distance is that we want to variance to be non-negative for further statistics concepts. On top of that, squared is preferred over absolute value since such a squared function is differentiable everywhere. Use np.var().

where $\mu = E[X]$.

Standard Deviation

Standard deviation of a random variable X, denoted $\sigma(X)$, is the weighted average of distance from the mean. Use np.std().

Note: (discussed more in Math for AI 106)

1. If outcomes are equally likely to happen and $N$ is the entire population’s size, we have

   \[E[X] = \frac{1}{N} \sum_{k} x_k \text{ and } var(X) = \frac{1}{N} \sum_{k} (x_k - \mu)^2.\]

2. If we only take a sample of size $n$, then N becomes $n - 1$. Add another argument ddof=1 when using NumPy.

Continuous Random Variable

Since PMFs for continuous random variables are always 0, we can only apply the idea to compute expected value, variance, and standard deviations rather than exactly the same formulas. Here, we use integral for expected value which is

This integral can be done using either Riemann’s sum or Lebesgue integral (and maybe other integrals). The former is applied for “nice” distribution while the latter can handle more complicated distributions. This is partly because the Lebesgue integral takes weighted coefficients into account better. Computers use the Lebesgue integral as its default method.

Covariance and Correlation

We’ve been dealing with cases with only one random variable. What if we’d like to know the relationship between two random variables?

This is where covariance and correlation come into play.

Covariance

Covariance measures the linear relationship of two random variables, that is, how they change together. Recall $n - 1$ is for samples as we consider $X$ and $Y$ as subset of the population. Use np.cov().

where $X$ and $Y$ are random variables such that $E[X] = \mu_X$ and $E[Y] = \mu_Y$.

We deduce from the above formula that $var(X) = cov(X, X)$. Formally, covariance is represented using the following 2-by-2 matrix

We can interpret the result as follows.

• $cov(X, Y) \in (-\infty, \infty)$.
• $cov(X, Y) = cov(Y, X)$.
• If it is positive covariance or $cov(X, Y) > 0$, then $X$ increases implies $Y$’s rise on average.
• If it is negative covariance or $cov(X, Y) < 0$, then $X$ reduces implies $Y$’s decrease on average.
• If it is zero covariance or $cov(X, Y) = 0$, then there is no linear correlation between two random variables.

DEMONATRATION

Correlation

Even though covariance is useful, its value can be unnecessarily large. In addition, covariance only indicates the direction of the relationship, not how much it is. To address this issue, we standardize the covariance to obtain the correlation coefficient.

Correlation is a statistical measure the quantifies the strength and direction of a linear relationship between two random variables. Use np.corrcoef().

It is also often represented as a 2-by-2 matrix like covariance.

We can interpret the result as follows.

• $corr(X, Y) \in [-1, 1]$.
• $corr(X, Y) = corr(Y, X)$.
• If it is positive correlation or $corr(X, Y) > 0$, we have as $X$ increases, $Y$’s rise on average.
  • If $corr(X, Y) = 1$, values of $X$ and $Y$ form a straight line with a positive slope.
  • If $corr(X, Y)$ is close to 1, then it is a strong relationship and values of $X$ and $Y$ lie close to a straight line.
  • If $corr(X, Y)$ is not so close to 1, then it is a weaker relationship and values of $X$ and $Y$ are distributed around but further away from a straight line.
• If it is negative covariance or $corr(X, Y) < 0$, we have as $X$ reduces, $Y$’s decrease on average.
  • If $corr(X, Y) = -1$, values of $X$ and $Y$ form a straight line with a negative slope.
  • If $corr(X, Y)$ is close to -1, then it is a strong relationship and values of $X$ and $Y$ lie close to a straight line.
  • If $corr(X, Y)$ is not so close to -1, then it is a weaker relationship and values of $X$ and $Y$ are distributed around but further away from a straight line.
• If it is zero covariance or $corr(X, Y) = 0$, then there is no linear relationship between two random variables and values scatter without a clear linear pattern.

DEMONATRATION

Continuous Random Variable

Like mathematical expectations, we cannot directly compute covariance and correlation of continuous random variables. In this case, we can convert continous random variables to discrete ones using Monte Carlo method. Softwares may have it set up in methods as follows.

# X ~ N(0, 1) 
# Y ~ N(0.5, 1.5)
data1 = np.random.normal(0, 1, 1000)
data2 = np.random.normal(0.5, 1.5, 1000)

# calculate covariance and correlation matrix
covariance_matrix = np.cov(data1, data2)
correlation_matrix = np.corrcoef(data1, data2)

FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.