Pandas Missing Values

Question 1

NaN (acronym for Not a Number)

Answer 1

The other missing data representation, NaN (acronym for Not a Number), is different; it is a special floating-point value recognized by all systems that use the standard IEEE floating-point representation:

Question 2

np.nansum(vals2), np.nanmin(vals2), np.nanmax(vals2)

Answer 2

NumPy does provide some special aggregations that will ignore these missing values:

Question 3

data.isnull()

Answer 3

Pandas data structures have two useful methods for detecting null data: isnull() and notnull(). Either one will return a Boolean mask over the data. For example

Question 4

data[data.notnull()]

Answer 4

Pandas data structures have two useful methods for detecting null data: isnull() and notnull(). Either one will return a Boolean mask over the data. For example

Question 5

data.dropna()

Answer 5

In addition to the masking used before, there are the convenience methods, dropna() (which removes NA values) and fillna() (which fills in NA values). For a Series, the result is straightforward:

Question 6

df.dropna(axis=’columns’)

Answer 6

Alternatively, you can drop NA values along a different axis; axis=1 drops all columns containing a null value:

Question 7

df.dropna(axis=’columns’, how=’all’)

Answer 7

The default is how=’any’, such that any row or column (depending on the axis keyword) containing a null value will be dropped. You can also specify how=’all’, which will only drop rows/columns that are all null values:

Question 8

df.dropna(axis=’rows’, thresh=3)

Answer 8

For finer-grained control, the thresh parameter lets you specify a minimum number of non-null values for the row/column to be kept:

Question 9

data.fillna(0)

Answer 9

We can fill NA entries with a single value, such as zero:

Question 10

# forward-fill
data.fillna(method='ffill')

# back-fill
data.fillna(method='bfill')

Answer 10

We can specify a forward-fill / back fill to propagate the previous value forward:

Question 11

df.fillna(method=’ffill’, axis=1)

Answer 11

or DataFrames, the options are similar, but we can also specify an axis along which the fills take place: