Pandas Series: Filtering, Selection and Indexing

Last updated: July 1st, 2019

Pandas Series: Filtering, selection and indexing¶

Pandas Series object acts in many ways like a one-dimensional NumPy array, and in many ways like a standard Python dictionary. If we keep these two overlapping analogies in mind, it will help us to understand the patterns of data indexing and selection in these data structures.

Hands on!¶

In [ ]:
import pandas as pd
import numpy as np


The first thing we'll do is create again the Series from our previous lecture:

In [ ]:
data_dic = {
'France': 63.951,
'Germany': 80.94,
'Italy': 60.665,
'Japan': 127.061,
'United Kingdom': 64.511,
'United States': 318.523
}

g7_pop = pd.Series(data_dic,
name='G7 Population in millions')

In [ ]:
g7_pop


Indexing¶

Indexing works similarly to lists and dictionaries.

Indexing by index¶

you use the index of the element you're looking for:

In [ ]:
g7_pop['Canada']

In [ ]:
g7_pop['Japan']

In [ ]:
g7_pop['United Kingdom']


The following also works, but it's NOT recommended:

In [ ]:
g7_pop.Japan


Slicing and multi-selection¶

Slicing also works, but important, in Pandas, the upper limit is also included:

In [ ]:
g7_pop['Germany': 'Japan']


Multi indexing also works (similarly to numpy):

In [ ]:
g7_pop[['Italy', 'France', 'United States']]


Indexing by sequential position¶

Indexing elements by their sequential position also works. In this case pandas evaluates the object received; if it doesn't exist as an index, it'll try by sequential position.

With sequential position the upper limit is not included.

In [ ]:
g7_pop

In [ ]:
g7_pop.iloc[0] # First element

In [ ]:
g7_pop.iloc[-1] # Last element


Other examples:

In [ ]:
g7_pop.iloc[2]

In [ ]:
g7_pop.iloc[4]

In [ ]:
g7_pop.iloc[2:4]

In [ ]:
g7_pop.iloc[[3, 1, 6]]


Adding new elements to a Series¶

In many cases we'll want to add new values to our Series, to do that we can just simply index our Series using the new index and then assigning a value to that index. Let's add two new records:

In [ ]:
g7_pop['Brazil'] = 20.124
g7_pop['India'] = 32.235

In [ ]:
g7_pop


Modifying Series elements¶

In [ ]:
g7_pop['Canada'] = 40.5

g7_pop

In [ ]:
g7_pop['France'] = np.nan

g7_pop


Removing elements from a Series¶

In [ ]:
del g7_pop['Brazil']

g7_pop

In [ ]:
del g7_pop['India']

g7_pop


Checking existance of a key (membership)¶

In [ ]:
'France' in g7_pop

In [ ]:
'Brazil' in g7_pop


Introducing loc & iloc¶

What's the problem with the indexing we've seen? It's not explicit. Pandas receives an element to index and it tries figuring out if we meant to select an element by its key, or its sequential position. Check out the following example:

In [ ]:
s = pd.Series(
['a', 'b', 'c'],
index=[1, 2, 3])
s

In [ ]:
s


What happens if we try indexing s[1], what should it return? a or b?

In [ ]:
s[1]


In this case, the returned object is worked out by the index, not by the sequential position. But again, it's not intuitive or explicit.

Enter loc and iloc:

• loc is the preferred way to select elements in Series (and Dataframes) by their index
• iloc is the preferred way to select by sequential position
In [ ]:
s.loc[1]

In [ ]:
s.iloc[1]

In [ ]:
g7_pop

In [ ]:
g7_pop.iloc[-1]

In [ ]:
g7_pop.iloc[[0, 1]]


Using our previous series:

In [ ]:
g7_pop

In [ ]:
g7_pop.loc['Japan']

In [ ]:
g7_pop.iloc[-1]

In [ ]:
g7_pop

In [ ]:
g7_pop.loc['Canada']

In [ ]:
g7_pop.iloc[0]

In [ ]:
g7_pop.iloc[-1]

In [ ]:
g7_pop.loc[['Japan', 'Canada']]

In [ ]:
g7_pop.iloc[[0, -1]]


loc & iloc to modify Series¶

In [ ]:
g7_pop.loc['United States'] = 1000

g7_pop

In [ ]:
g7_pop.iloc[-1] = 500

g7_pop


Introducing to Boolean arrays¶

Another way to select certain values within a Series is using boolean arrays, also known as Conditional selection.

We can index our Series using a list of boolean values:

In [ ]:
g7_pop[[False, False,  True, False,  True, False,  True]]


Or we can index our Series using another Series with boolean values:

In [ ]:
condition = pd.Series([
False, False,  True, False,  True, False,  True
], index=[
'Canada', 'France', 'Germany', 'Italy', 'Japan', 'United Kingdom', 'United States'
])

condition

In [ ]:
g7_pop[condition]


On next lecture we'll see how to use more complex conditional selections.