Data: the final frontier

Execute the following cell only if you need to install the seaborn library

!pip install --upgrade matplotlib seaborn

The following libraries are the foundation tools:

• pandas is an in-memory dataframe library
• matplotlib is a plotting library inspired by matlab plotting API
• seaborn is a chart library based on matplotlib, with more functionalities and themes
• numpy is a numeric calculation library providing fast c arrays and scientific functions

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
from tabulate import tabulate

let’s start with a classic dataset including the passengers of Titanic ship.

The read_csv function uploads this format in a pandas DataFrame which is a relation

Note: the titanic dataset was downloaded at the beginning of Part 2; in case you missing it execute the code at the beginning of the lesson

The .head() method returns the first lines of your data frame to quickly inspect it

titanic = pd.read_csv("datasets/titanic.csv")
df =titanic.head()[["Survived","Pclass","Age","Sex"]]

the .describe() method returns basic statistics for all numerical columns

• min
• max
• median
• mean
• quartiles
• count of elements

by using the .describe(include“all”)= option also categorical values are shown with some other statistics:

• number of unique discrete values
• the most common one
• its frequency

df =titanic.describe(include="all")[["Survived","Pclass","Age","Sex"]]

It is possible to access columns (called Series in pandas jargon) using the square bracket operator

titanic["Pclass"]

columns whose name is a good python identifier (i.e. starts with a letter and contains only letters, numbers and underscore) can be accessed using the dot notation e.g.

titanic.Pclass

each column has a data type, as csv do not carry any type information, this is inferred when loading; other binary data format also include a data type. The datas type of a column is saved in the .dtype attribute

pclass = titanic.Pclass
print(pclass.dtype)

int64

we know this column represents the class of the ticket so we expect it to have a finite number of actual values: we can check it with the .unique() method

df =pclass.unique()

we see this is a discrete valued columns so we can transform its type with the .astype() method

pclass = pclass.astype('category')
df =pclass.dtype

Now the statistics are represented differently for pClass

titanic["pClass"] = pclass
df =titanic.describe(include="all")[["Survived","Pclass","Age","Sex"]]

If we know in advance about the type of a column we can give some hint to the csv reader

titanic = pd.read_csv(
    "datasets/titanic.csv",
    dtype={
        "Survived":"category",
        "Pclass":"category",
        "Sex":"category",
    }
)

When we have a category series we can list all of the possible values using the .cat.categories attribute

print(pclass.cat.categories)

Index([1, 2, 3], dtype=’int64’)

the sns.countplot() function show a bar plot of categorical values

sns.countplot(pclass)

this dataframe collects pollutant density in California

california = pd.read_csv("california_pb_2023.csv")
df =california.describe(include="all")[['Daily Mean Pb Concentration', 'County']]

sns.histplot shows an histogram

sns.histplot(california,x="Daily Mean Pb Concentration")

This distribution looks like a lognormal distribution, let’s show a cumulative distribution and plot it with a logaritmic x axis

sorted_pb = np.sort(california["Daily Mean Pb Concentration"])
prob_pb = (np.arange(len(sorted_pb)) + 1)/len(sorted_pb)
ax=sns.lineplot(x=sorted_pb, y=prob_pb)
ax.set_xscale("log", base=10)

This looks nice so we can check by fitting a quantile plot

First we try with a normal quantile, we expect some queues

from scipy import stats
stats.probplot(california["Daily Mean Pb Concentration"], plot=sns.mpl.pyplot)

We can fit it with a different distribution, so we choose a lognormal

stats.probplot(california["Daily Mean Pb Concentration"], plot=sns.mpl.pyplot,dist=stats.distributions.lognorm(s=1))

this looks quite better

let’s consider a group of categorical variables and explore their interaction, the pd.crosstab() function provides a way to create a contingency table i.e. a table which counts all combination of the considered factors

titanic['survived'] = titanic.Survived.astype('category')
titanic['sex'] = titanic.Sex.astype('category')
titanic['pclass'] = titanic.Pclass.astype('category')
ct = pd.crosstab(titanic['survived'],columns=[titanic['sex'],titanic['pclass']])
df =ct

the .plot.bar() method provides a quick way to display this information as grouped bar plot

ct.plot.bar()

ct.plot.bar(stacked=True)

the iris dataset is a collection of measurements of this flower’s features (sepal and petal length and width) across different varieties.

iris = pd.read_csv("iris.csv")
df =iris.head()