Missing Data¶

Hands on!¶

What does "missing data" mean? What is a missing value? It depends on the origin of the data and the context it was generated. For example, for a survey, a Salary field with an empty value, or a number 0, or an invalid value (a string for example) can be considered "missing data". These concepts are related to the values that Python will consider "Falsy":

For Python, all the values above are considered "falsy":

Numpy has a special "nullable" value for numbers which is np.nan. It's NaN: "Not a number"

The np.nan value is kind of a virus. Everything that it touches becomes np.nan:

This is better than regular None values, which in the previous examples would have raised an exception:

For a numeric array, the None value is replaced by np.nan:

As we said, np.nan is like a virus. If you have any nan value in an array and you try to perform an operation on it, you'll get unexpected results:

Numpy also supports an "Infinite" type:

Which also behaves as a virus:

Checking for nan or inf¶

There are two functions: np.isnan and np.isinf that will perform the desired checks:

And the joint operation can be performed with np.isfinite.

np.isnan and np.isinf also take arrays as inputs, and return boolean arrays as results:

Note: It's not so common to find infinite values. From now on, we'll keep working with only np.nan

Filtering them out¶

Whenever you're trying to perform an operation with a Numpy array and you know there might be missing values, you'll need to filter them out before proceeding, to avoid nan propagation. We'll use a combination of the previous np.isnan + boolean arrays for this purpose:

Which is equivalent to:

And with that result, all the operation can be now performed: