Data Visualization and Analysis for AI 101

Acknowledgement: This post is based mainly on Dr. Quang-Vinh Dinh’s presentation and slides and partly on on TA, STA, and peers’ responses from the course AIO2025 offered by AI VIETNAM.

Relevance: The audience of this lecture were those who have read Math for AI 106, had some coding experience, and known a bit of AI knowledge.

In today’s post, we’ll explore pandas Series and DataFrame. We’ll also discuss some applications in AI including processing missing values, noise reduction, and data visualization.

pandas was built on top of NumPy, specifically for data analysis and visualization, like Excel. Since it has NumPy as an underlying structure, we can use what we know about NumPy to apply here.

pandas Series

pandas Series is a 1D data array like a one-column table. Upone its creation, the numerical indices are automatically generated by default and shown in the rendered output, but they as a whole are not considered as a column. We can update the indices as desired—that is, an index can be any real number, a letter, or other data types.

Series Creation

Create a Series from a List

data1 = pd.Series([2, 5, 100, 7, 3, 4], name='score')
data1

'''
0      2
1      5
2    100
3      7
4      3
5      4
Name: score, dtype: object
'''

data2 = pd.Series(['C++', 'Java', 'Python', 'Golang', 'Swift'], 
                 index=list('CJPGS'),
                 name='Programming Language')
data2

'''
C       C++
J      Java
P    Python
G    Golang
S     Swift
Name: Programming Language, dtype: object
'''

Get Elements

Use data1. Just like ndarray in NumPy, we can apply indexing and slicing for pandas Series to get one or more elements.

# values in a Series is an ndarray
data1.values
>>> array([  2,   5, 100,   7,   3,   4], dtype=int64)

# indexing
data1[1]
>>> 5

# slicing
data1[2:4]
>>> 2   100
    3     7
    Name: score, dtype: object

A list of indices can be passed as an argument to obtain information from specific rows that may be unevenly-spaced.

data2[['C', 'J', 'C']]
>>> C     C++
    J    Java
    C     C++
    Name: Programming Language, dtype: object

pandas has other methods to obtain values in a Series like loc which may be more non-tech-friendly.

data1.loc[2:4]
>>> 2   100
    3     7
    4     3
    Name: score, dtype: object

Similarly, pandas also has boolean indexing.

# boolean indexing
data1[data1 < 4]
>>> 0    2
    4    3
    Name: score, dtype: int64

# boolean indexing
data1[data1.between(5, 7)]
>>> 1    5
    3    7
    Name: score, dtype: int64

For data2 with non-numeric indices, we can either use numeric or customized indices to retrieve elements.

data2['C']
>>> 'C++'

data2.loc['C':'J']
>>> C     C++
    J    Java
    Name: Programming Language, dtype: object

Operations

Drop a Row
We can remove an existing row by using drop() with indexing. This method returns a new Series instead of changing the original. Note that the returned Series keeps the original indices.

data1.drop([1, 3])
>>> 0      2
    2    100
    4      3
    5      4
    Name: score, dtype: int64

data1
>>> 0      2
    1      5
    2    100
    3      7
    4      3
    5      4
    Name: score, dtype: object

Insert a Row
Inserting a Row in pandas is similar to that for a Python dictionary. The new row is ed to the original Series or changes an existing row if they have the same index.

data2['K'] = 'Kotlin'
data2
>>> C       C++
    J      Java
    P    Python
    G    Golang
    S     Swift
    K    Kotlin
    Name: Programming Language, dtype: object

If an index is not an integer, we can certainly reset the Series so that it is properly indexed. This method does not change the original Series.

data1[2.5] = 99

# without drop=True, the old index column is ed
data1.reset_index(drop=True)
>>> 0      2
    1      5
    2      100
    3      7
    4      3
    5      4
    6      99
    Name: score, dtype: int64  

Sort a Series
Sorting a Series by its index can be done with method sort_index(). We can decide if the original Series should be updated or not by the argument inplace, in particular,

inplace=True returns None and update the original Series.
inplace=False returns a sorted Series and does not update the original Series.

# update the original Series
data2.sort_index(inplace=True)
data2
>>> C       C++
    G    Golang
    J      Java
    K    Kotlin
    P    Python
    S     Swift
    Name: Programming Language, dtype: object

# NOT update the original Series
sorted_data2 = data2.sort_index(inplace=False)
sorted_data2 
>>> C       C++
    G    Golang
    J      Java
    K    Kotlin
    P    Python
    S     Swift
    Name: Programming Language, dtype: object

data2
>>> C       C++
    J      Java
    P    Python
    G    Golang
    S     Swift
    K    Kotlin
    Name: Programming Language, dtype: object

Common Functions
pandas contains NumPy’s common functions

series.min()
series.max()
series.idxmin() or series.argmin()
series.idxmax() or series.argmax()
series.sum()
series.mean()
series.var()
series.std()
series.str.count(): the argument can be a character or a string and it counts the number of characters or strings in each value.
series.str.upper()
series.replace(current_value, new_value): replace the current value with a new value.

** Two Series**
If values from Series are of the same data type, they can be ed or concatenated. Behind the scenes, pandas looks for the matching index and two values together. If it cannot find a row’s matching index, it will render NaN.

# modify data1 to have 6 rows
data1[2.5] = 99
data1.sort_index(inplace=True)

# create data3 of 5 rows
data3 = pd.Series([10, 10, 10, 10, 10])

#  data1 and data3
data1 + data3
>>> 0.0     12.0
    1.0     15.0
    2.0    110.0
    2.5      NaN
    3.0     17.0
    4.0     13.0
    5.0      NaN
dtype: float64

Group Values
If some values have the same name/index and we want to group them together, we can use groupby(). This syntax returns another Series, so we can certainly use Series methods or function on it.

data4 = pd.Series([4, 1, 0, 2, 2, 1, 8],
                  index=list(['C++', 'Golang', 'Java', 'Golang', 'Java', 'C++', 'Python']),
                  name='groupby')

# test
data4.groupby(level=0).sum()
>>> C++         5
    Golang      3
    Java        2
    Python      8
    Name: groupby, dtype: int64

data4.groupby(data4 > 3).sum()
>>> groupby
    False     6
    True     12
    Name: groupby, dtype: int64

data4['Golang'].count()
>>> 2

Application in AI

1D Interpolation

The problem introduced here appears in scaling images and the two techniques are a part of Super-resolution Imaging. Since scaling an image increases the number of cells but newly ed cells need values.
Another situation is when we work with missing data points. We can either remove or keep them depending on the type of datasets. For those that stay, we need to make sure they have values. For example, time series data can be related with each other, so we’d like to keep missing values.

In both cases, we need to fill in context-appropriate rather than random values. For some datasets, we fill in with the average. In ition, we use interpolate() with a method argument to resolve this issue.

Linear Interpolation: method='linear' by default.
Nearest Neighbor Interpolation: method='nearest'.

data5 = pd.Series([2, 5, 100, np.nan, 7, 3, np.nan, 9, np.nan], name='score')

# linear
data5.interpolate(method='linear')

# nearest neighbor
data5.interpolate(method='nearest')

In the example above, pandas considered 100 and 7 for the first np.nan.

original data with NaN — Original data with NaN

data after applying linear interpolation — Original data with NaN

Nearest Neighbor Interpolation vs. Linear Interpolation

Problem statement: Assume there are two data points, find another data point given one of its coordinates.
Breakdown:

$x$ is the index, $y$ is the value in a Series.
$(x_1, y_1)$ and $(x_2, y_2)$: two given data points.
$x_3$: known coordinate of the third data point.
$y_3$: unknown coordinate of the third data point.

Idea: There are two simple but effective techniques: nearest neighbor interpolation and linear interpolation.

	Nearest Neighbor	Interpolation
Solution	Compute distance from $x_3$ to $x_1$ and $x_2$. Pick the y-value of the smaller one.	Find the linear equation passing $(x_1, y_1)$ and $(x_2, y_2)$. Take the corresponding y-value on that line. $$\begin{align} &y_3 = \frac{x_2 - x_3}{x_2 - x_1} y_1 \\ &+ \frac{x_3 - x_1}{x_2 - x_1} y_2 \end{align}$$
Application	If objects in an image are straight up perpendicular to the edges, they should look fine when the image is scaled. If they are slanted, they could look like a floor function, Lego blocks, or images in Minecraft.	This can be better for slanted objects.

image scaling using nearest neighbor technique — Image scaling using nearest neighbor technique

Figure 2. $f(x)$ represents the nearest neighbor interpolation. $g(x) = ax + b$ represents linear interpolation. Move the sliders for points $A$ and $B$ to set their y-values, then drag the slider point along the x-axis to choose the input $x$. The applet updates $f(x)$ and $g(x)$ to show the estimated values using each methodView the interactive applet. Click on the symbol in the bottom right corner to open the applet in full-screen mode. If the applet does not load, you can open it directly here.

Noise Reduction
Noise refers to the random irregularity or fluctuations in real-life datasets. There are a few ways to reduce noise in statistics.

Simple Moving Average (SMA): the simplest way in which we take the mean of data windows.

\[SMA_t = \frac{s_t + \cdots + s_{t-k+1}}{k}\]

where $k$ is the size of each window.

Exponential Moving/Weighted Average (EMA): widely used in optimization, considering both the previously computed values and the current value.

\[EMA_t = \rho EMA_{t-1} + (1 - \rho) s_t\]

Later on, we’ll mainly use EMA, but for now let’s look at the graph of the above dataset after using SMA. The larger the window is, the smoother the graph is.

linear_sma_5 = linear_int.rolling(window=5).mean()
linear_sma_5.head(100).plot()

Practice Exercises

Exercise 1: For data2, what is the output of data2['D'] and data2['C':'X']? Explain why.
Exercise 2: Work out the formulae for linear interpolation technique.
Exercise 3: Which values does pandas consider for second and third np.nan in each case of the interpolation example?
Exercise 4: Write code for the two other windows of the SMA example (linear and nearest neighbor).

DataFrame

A DataFrame is a 2D labeled data structure. Unlike a Series, a DateFrame can have more than one data column. Basically, it is a combination of multiple Series. Thus, we can apply methods/functions in Series to work with DataFrames.

Get Values
We can also apply indexing and slicing for pandas Series to get one or more elements like a NumPy array—using a comma to separate axes.

df = pd.read_csv('advertising_simple.csv')
df
>>>     TV  Radio   Newspaper   Sales
    0   44     39          45      10
    1   17     45          69      12
    2   151    41          58      16
    3   180    10          58      17
    4   8      48          75      7
    5   57     32          23      11
    6   120    19          11      13
    7   8      2           1       4

# get 'Radio' column
df.loc[:, 'Radio']
>>> 0     39
    1     45
    2     41
    3     10
    4     48
    5     32
    6     19
    7      2
    Name: Radio, dtype: int64

# get a DataFrame of 'Radio' column
df.loc[:, ['Radio']]
>>>    Radio
    0     39
    1     45
    2     41
    3     10
    4     48
    5     32
    6     19
    7      2

# another way to get 'Radio' column
df.Radio
>>> 0     39
    1     45
    2     41
    3     10
    4     48
    5     32
    6     19
    7      2
    Name: Radio, dtype: int64

Besides loc, pandas has iloc to get values using index rather than label. Similar to NumPy, pandas has boolean indexing.

df.loc[df.Sales > 10]
>>>     TV  Radio   Newspaper   Sales
 17     45          69      12
 151    41          58      16
 180    10          58      17
 57     32          23      11
 120    19          11      13

Sort Values
pandas lets us sort data based on one or multiple columns sequentially.

# sort by one column
df.sort_values(['Newspaper'])

# sort by more than one column
# 'Newspaper' first, then based on that, 'Sales'
df.sort_values(['Newspaper', 'Sales'])

Inser a Column
We can insert a column to a DataFrame like ing an item in Python dictionary. The original DataFrame is updated. Besides the default index ‘colum’, we can certainly set a column to be indices. Note that setting indices does not affect the original.

#  'ID' column
df['ID'] = list(range(2, 10))

# set index
df = df.set_index('ID')
df
>>>     TV  Radio   Newspaper   Sales
    ID				
    2   44     39          45      10
    3   17     45          69      12
    4   151    41          58      16
    5   180    10          58      17
    6   8      48          75      7
    7   57     32          23      11
    8   120    19          11      13
    9   8      2           1       4

Concatenate Rows
If we want to combine two DataFrames with the same columns together, we use concat([dataFrame1, dataFrame2]). Since the original indices from each DataFrame remain unchanged, we may need to reset index and drop the old one by reset_index(drop=True).

Datetime
When data is collected, time is often inserted as strings, which makes it harder for computers to understand and analyze. Thus, it’s recommended time data be converted to real datetime format. Use to_datetime() method.

df = pd.read_csv('WeatherHistory2D.csv')

# convert string to datetime format
time_col = 'Formatted Date'
df[time_col] = pd.to_datetime(df[time_col],
                              utc=True)

# get each feature of datetime 
#  a column for each feature
df['Year'] = df[time_col].dt.year
df['Month'] = df[time_col].dt.month
df['Day'] = df[time_col].dt.day
df['Hour'] = df[time_col].dt.hour

Remove a Column
In the example above, the “Formatted Date” column is no longer needed as we separate columns for datetime features. We can just delete it using df.drop(label, axis). This syntax returns a new DataFrame rather than changing the original. We can certainly delete multiple columns.

df.drop(time_col, axis=1)
>>>     Temperature (C) Year    Month   Day Hour
    0      0.577778     2005       12    31   23
    1      1.161111     2006       1     1    0

Group Values
We can certain group values by columns. That is, if there are duplicate rows in a column, we can combine them together and use functions to analyze each of them as a whole.

df_group = pd.read_csv('weatherHistory_simple.csv')
df_group
>>>     Precip Type Temperature (C)
    0          rain            12.2
    1          rain            13.8
    2          rain            14.7
    3          rain            13.8
    4          rain            12.7
    5          rain            0.5
    6          snow            -0.4
    7          snow            -1.1
    8          snow            -1.6
    9          snow            -2.1

# group by 'Precip Type' column and compute mean of each feature by 'Temperature (C)' column
df_group.groupby('Precip Type').mean('Temperature (C)')
>>>         Temperature (C)
Precip Type	
rain              11.283333
snow              -1.300000

Convert Categories to Numbers
In the example above, the “Precip Type” column can be a class column as in a classification problem. However, computers can only work with numeric values, it’s better that we convert those string categories into numbers.

# test
col_to_num = 'Precip Type'
pd.Categorical(df_group[col_to_num])
>>> [0, 0, 0, 0, 0, 0, 1, 1, 1, 1]
    Categories (2, int8): [0, 1]

pd.Categorical(df_group[col_to_num]).codes
>>> array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1], dtype=int8)

# convert categories to numbers
col_to_num = 'Precip Type'
df_group[col_to_num] = pd.Categorical(df_group[col_to_num]).codes
df_group
>>>     Precip Type Temperature (C)
    0             1           12.2
    1             1           13.8
    2             1           14.7
    3             1           13.8
    4             1           12.7
    5             1           0.5
    6             0           -0.4
    7             0           -1.1
    8             0           -1.6
    9             0           -2.1

Descriptive Statistics
We can check out all the basic information related to a dataset by info(). pandas also calculates basic descriptive statistics and lets us see them via describe() method.

df_weather.loc[:, ['Temperature (C)']].describe()
>>>         Temperature (C)
    count      96453.000000
    mean       11.932678
    std        9.551546
    min        -21.822222
    25%        4.688889
    50%        12.000000
    75%        18.838889
    max        39.905556

The numbers $25\%$, $50\%$, and $75\%$ are called k-th percentiles where $k$ is the percentage.

$50\%$ is the median.
$25\%$ means one fourth of the sorted dataset from minimum.
$75\%$ means one fourth of the sorted dataset from maximum.

Using percentiles, we can indicate if the distribution of a dataset is more left-skewed or right-skewed.

Pro tip: pandas includes many other functions and methods that allows us to analyze how training models work for a dataset. This helps us know which model we should use for our datasets in certain cases.

Practice Exercises

Exercise 1: What is the difference between df.loc[:, 'Radio'] and df.loc[:, ['Radio']] in the indexing and slicing part above?
Exercise 2: Write code to get the "Radio" column without using loc.
Exercise 3: Write code to get rows 1-4 using the output of exercises 1 and 2.
Exercise 4: Write code to drop "Radio" and "Sales" columns in exercise 1.
Exercise 5: Write code to drop rows 1, 3, 6 in exercise 1 (before and after setting index).
Exercise 6: For examples containing 'Temperature (C)', write code to add a 'Temperature (F)' column.
Exercise 7: Using data in exercise 1, what is the output of df.loc[3, 'Radio'], df.loc[3, 4], df.loc[2::2], df.drop('Radio'), and df.iloc[1:4, 1:5]?

Visualization

We can use seaborn, which is built on top of matplotlib to visualize data.

# set up seaborn
import seaborn as sns

We’ll use a famous Iris flower classification example here.

df_iris = pd.read_csv('iris.csv')

# graph two columns in one figure
sns.jointplot(df_iris, x='PetalLength', y='PetalWidth', hue='Name')

This kind of plot is quite beneficial for both classification and prediction problems when we analyze to choose the right columns to compare.

sepal length vs sepal width — Sepal Length vs Sepal Width

petal length vs petal width — Sepal Length vs Sepal Width

We can use pairplot() to see all the pairs.

plots for every numerical pairs — *Figure 5.*Plots for every numerical pairs

We can also check out the distribution of each type as follows. Boxplot gives the statistical structure of a feature where min, three primary percentiles, and max are shown from top to bottom by default.

# dotplot 
sns.stripplot(df_iris, y='SepalLength', x='Name', hue='Name')

# boxplot
sns.boxplot(df_iris, y='SepalLength', x='Name', hue='Name', width=0.3)

sepal length dotplot — Sepal Length dotplot

sepal length boxplot — Sepal Length dotplot

Practice Exercises

Exercise 1: Write code to plot the rest for the Iris flower example give 5 columns as above.

FAQ

Here are some good questions asked during class. The answers to these questions came from either some admin, TA, STA, or members of AIO2025.

Questions	Answers
Is there a way to interpolate for one column using other columns?	Yes. There are many ways, for example using decision tree, random forest.
How do I know which method I should use to fill missing values?	We try, particularly from the baseline method, interpolation, then improve it.
How do I know if a dataset has noise?	Ask experts in that field.
After filling in missing values, data will be distorted. Is it okay?	Yes. Actually we want to distort data properly since we care more about the data from the global and long-term point of view.
If a feature X has 2 missing values belonging to 2 different classes, how should we go about filling in those values?	You can fill them according to values in their own class or use other features to decide.

pandas Series

Series Creation

Operations

Application in AI

DataFrame

Visualization

FAQ

Enjoy Reading This Article?