1.3. Probability

Last updated: May 26th, 20202020-05-26Project preview
In [2]:
import pandas as pd

Probability

The two random experiments we have performed so far as examples were to pick a person and print their city, and to pick a person and print their age:

In [3]:
dataset =  pd.DataFrame({
    'Person #':[1,2,3,4,5,6,7,8,9,10],
    'City':['SF','SF','NY','NY','NY','SF','NY','SF','SF','SF'],
    'Age':[41,26,28,53,32,51,65,49,25,33]
})
dataset
Out[3]:
Person # City Age
0 1 SF 41
1 2 SF 26
2 3 NY 28
3 4 NY 53
4 5 NY 32
5 6 SF 51
6 7 NY 65
7 8 SF 49
8 9 SF 25
9 10 SF 33
In [4]:
dataset.sample(1)['City'].values[0]
Out[4]:
'SF'
In [5]:
dataset.sample(1)['Age'].values[0]
Out[5]:
26

Now, even though in both cases we cannot be certain about the outcome, we can clearly see that getting 'NY' is much more likely than getting '51'.

The likelihood of an outcome is what we call the theoretical probability of that outcome.

Formally, we will write $P(\omega)$ for $\omega\in\Omega$.

When we perform an experiment, we can assign this theoretical probability to each outcome, for example the probability of getting heads when we toss a fair coin is $1/2$.

In following videos we will learn how to calculate these probabilities for different problems.

But if we only have data it can be difficult to know exactly what the theoretical probability of a given outcome is.

So another way of defining the probability of an outcome is by performing the random experiment a large number of times and calculating the proportion of times we get our outcome, also known as the relative frequency of that outcome.

This is what we call the empirical probability of the outcome.

Let's explore this with a bigger dataset:

In [6]:
toy_dataset = pd.read_csv('toy_dataset.csv')
toy_dataset.head()
Out[6]:
Number City Gender Age Income Illness
0 1 Dallas Male 41 40367.0 No
1 2 Dallas Male 54 45084.0 No
2 3 Dallas Male 42 52483.0 No
3 4 Dallas Male 40 40941.0 No
4 5 Dallas Male 46 50289.0 No

In this dataset we not only have the city and age of each person, but also their gender, income and whether they have an illness.

In [7]:
toy_dataset.shape
Out[7]:
(150000, 6)

We can also check that the dataset has $150000$ rows (or people).

So let's explore the notion of empirical probability by carrying out an experiment a large number of times (for example $100$), and calculating the proportion of times we get a particular result, for example picking a person and the gender being 'Female':

In [12]:
data = toy_dataset.sample(100, replace=True)['Gender'].value_counts().to_frame()
data
Out[12]:
Gender
Male 59
Female 41
In [13]:
proportion = data.loc['Female'][0]/100
proportion
Out[13]:
0.41

We can see, by repeatedly running this code, that this proportion constantly changes. So we could ask ourselves, what is a large number of times?

In theory, if we could perform the experiment infinitely many times, we would arrive at the theoretical probability (actually this is a result that we will see in the last course).

Of course in real life there is no way we can perform an experiment infinitely many times, but let's try to get an intuition of this result, by comparing the proportions we get when we perform the experiment an increasing number of times:

In [15]:
times = [100,1000,10000,100000]

proportions = {}

for t in times:
    experiments = []
    for experiment_id in range(10):
        data = toy_dataset.sample(t, replace=True)['Gender'].value_counts().to_frame()
        experiments.append(data.loc['Female'][0] / t)
    proportions[f"Times: {t}"] = experiments
    
results = pd.DataFrame(proportions)
results.index = ['Experiment %s' % i for i in range(1, 11)]

results
Out[15]:
Times: 100 Times: 1000 Times: 10000 Times: 100000
Experiment 1 0.52 0.450 0.4444 0.43985
Experiment 2 0.43 0.447 0.4432 0.44176
Experiment 3 0.47 0.462 0.4533 0.44039
Experiment 4 0.34 0.421 0.4488 0.44127
Experiment 5 0.47 0.476 0.4498 0.44203
Experiment 6 0.42 0.421 0.4338 0.44194
Experiment 7 0.45 0.447 0.4414 0.44383
Experiment 8 0.41 0.459 0.4344 0.44082
Experiment 9 0.55 0.414 0.4489 0.44192
Experiment 10 0.46 0.450 0.4388 0.44351

What we are doing here is calculating the proportion of 'Female' we get when performing of the experiment $100$ times, $1000$ times, $10000$ times and $100000$ times, and making $10$ repetitions of each to compare the results.

The code builds a dictionary that maps the amount of times we are performing the experiment with a list of the $10$ proportions we get each time we do that.

What we can see is that the more times we perform the experiment, the more consistent the proportion gets, showing that it is approaching the theoretical probability when the amount of times we perform the experiment grows towards infinity.

Now it's your turn to explore some relative frequencies to estimate probabilities!

Notebooks AI
Notebooks AI Profile20060