Profile picture

Pandas Series: Filtering, Selection and Indexing

Last updated: July 1st, 20192019-07-01Project preview

rmotr


Pandas Series: Filtering, selection and indexing

Pandas Series object acts in many ways like a one-dimensional NumPy array, and in many ways like a standard Python dictionary. If we keep these two overlapping analogies in mind, it will help us to understand the patterns of data indexing and selection in these data structures.

purple-divider

Hands on!

In [ ]:
import pandas as pd
import numpy as np

green-divider

The first thing we'll do is create again the Series from our previous lecture:

In [ ]:
data_dic = {
    'Canada': 35.467,
    'France': 63.951,
    'Germany': 80.94,
    'Italy': 60.665,
    'Japan': 127.061,
    'United Kingdom': 64.511,
    'United States': 318.523
}

g7_pop = pd.Series(data_dic,
                   name='G7 Population in millions')
In [ ]:
g7_pop

green-divider

Indexing

Indexing works similarly to lists and dictionaries.

Indexing by index

you use the index of the element you're looking for:

In [ ]:
g7_pop['Canada']
In [ ]:
g7_pop['Japan']
In [ ]:
g7_pop['United Kingdom']

The following also works, but it's NOT recommended:

In [ ]:
g7_pop.Japan

 Slicing and multi-selection

Slicing also works, but important, in Pandas, the upper limit is also included:

In [ ]:
g7_pop['Germany': 'Japan']

Multi indexing also works (similarly to numpy):

In [ ]:
g7_pop[['Italy', 'France', 'United States']]

 Indexing by sequential position

Indexing elements by their sequential position also works. In this case pandas evaluates the object received; if it doesn't exist as an index, it'll try by sequential position.

With sequential position the upper limit is not included.

In [ ]:
g7_pop
In [ ]:
g7_pop.iloc[0] # First element
In [ ]:
g7_pop.iloc[-1] # Last element

Other examples:

In [ ]:
g7_pop.iloc[2]
In [ ]:
g7_pop.iloc[4]
In [ ]:
g7_pop.iloc[2:4]
In [ ]:
g7_pop.iloc[[3, 1, 6]]

green-divider

Adding new elements to a Series

In many cases we'll want to add new values to our Series, to do that we can just simply index our Series using the new index and then assigning a value to that index. Let's add two new records:

In [ ]:
g7_pop['Brazil'] = 20.124
g7_pop['India'] = 32.235
In [ ]:
g7_pop

green-divider

Modifying Series elements

In [ ]:
g7_pop['Canada'] = 40.5

g7_pop
In [ ]:
g7_pop['France'] = np.nan

g7_pop

green-divider

Removing elements from a Series

In [ ]:
del g7_pop['Brazil']

g7_pop
In [ ]:
del g7_pop['India']

g7_pop

green-divider

 Checking existance of a key (membership)

In [ ]:
'France' in g7_pop
In [ ]:
'Brazil' in g7_pop

green-divider

Introducing loc & iloc

What's the problem with the indexing we've seen? It's not explicit. Pandas receives an element to index and it tries figuring out if we meant to select an element by its key, or its sequential position. Check out the following example:

In [ ]:
s = pd.Series(
    ['a', 'b', 'c'],
    index=[1, 2, 3])
s
In [ ]:
s

What happens if we try indexing s[1], what should it return? a or b?

In [ ]:
s[1]

In this case, the returned object is worked out by the index, not by the sequential position. But again, it's not intuitive or explicit.

Enter loc and iloc:

  • loc is the preferred way to select elements in Series (and Dataframes) by their index
  • iloc is the preferred way to select by sequential position
In [ ]:
s.loc[1]
In [ ]:
s.iloc[1]
In [ ]:
g7_pop
In [ ]:
g7_pop.iloc[-1]
In [ ]:
g7_pop.iloc[[0, 1]]

Using our previous series:

In [ ]:
g7_pop
In [ ]:
g7_pop.loc['Japan']
In [ ]:
g7_pop.iloc[-1]
In [ ]:
g7_pop
In [ ]:
g7_pop.loc['Canada']
In [ ]:
g7_pop.iloc[0]
In [ ]:
g7_pop.iloc[-1]
In [ ]:
g7_pop.loc[['Japan', 'Canada']]
In [ ]:
g7_pop.iloc[[0, -1]]

loc & iloc to modify Series

In [ ]:
g7_pop.loc['United States'] = 1000

g7_pop
In [ ]:
g7_pop.iloc[-1] = 500

g7_pop

green-divider

Introducing to Boolean arrays

Another way to select certain values within a Series is using boolean arrays, also known as Conditional selection.

We can index our Series using a list of boolean values:

In [ ]:
g7_pop[[False, False,  True, False,  True, False,  True]]

Or we can index our Series using another Series with boolean values:

In [ ]:
condition = pd.Series([
    False, False,  True, False,  True, False,  True
], index=[
    'Canada', 'France', 'Germany', 'Italy', 'Japan', 'United Kingdom', 'United States'
])

condition
In [ ]:
g7_pop[condition]

On next lecture we'll see how to use more complex conditional selections.

purple-divider

Notebooks AI
Notebooks AI Profile20060