Data Preprocessing/Cleaning

Question 1

How do you print the first 20 rows of a dataframe called df?

Answer 1

df.head(20)

Question 2

What does the .info() method called on a pandas dataframe show?

Answer 2

The object type, column names data types, size of dataframe in memory, column counts, and range of row index.

Question 3

How do you print the column names of a pandas dataframe, df?

Answer 3

df.columns

Question 4

How do you iterate over columns of a dataframe, df, and remove spaces?

Answer 4

df.columns.str.replace(“ “, “”)

Question 5

How do you return basic summary statistics for the continuous variables in a pandas dataframe, df, and what does this method show?

Answer 5

df.describe():

Row count
Mean
Std
Min
25% Quantile
50% Quantile
75% Quantile
Max

Question 6

How do you print a column (series) of a dataframe, df?`

What does this command also return?

Answer 6

df[‘column name’]
df.column_name

This will also return the number of rows as length and the datatype.

Question 7

How do you change the data type of a pandas dataframe, df, column?

Answer 7

df.column.astype(‘x’)

Question 8

Use a lambda function to remove the left whitespace of all rows of dataframe column’s categories, df, and remove all periods from category names

Answer 8

df.column.apply(lambda x: x.lstrip().replace(“.”,””))

Question 9

How would you show how many cars were manufactured for each origin country in the cars dataset, for every year?

Answer 9

pd.crosstab(cars.year, cars.brand)

Question 10

What are two methods for returning rows and columns from a dataframe, df, with a specified condition?

Answer 10

df. loc[]

df. iloc[]

Question 11

How do you return the sum of all NaN values from columns in a dataframe, df?

Answer 11

df.isna().sum()

Question 12

How do you fill all of the NaN values of the cylinders column with the rounded mean of that column?

Answer 12

cars.cylinders.fillna(round(np.nanmean(cars.cylinders)), inplace = True)

Question 13

How do you calculate the mode for a categorical column, brand, excluding “Missing” category for the brand column?

Then how do you replace all of the “Missing” values with the computed mode value?

Then how do you remove the “Missing” category?

Answer 13

mode_brand = cars.brand[cars.brand != “Missing”].mode()

cars. brand[cars.brand == “Missing” = mode_brand[0]
cars. brand.cat.remove_unused_categories()

Question 14

How do you calculate the min and max of a pandas dataframe, df, using numpy?

Answer 14

np. min(df.column)

np. max(df.column)

Question 15

How do you perform min-max normalization?

Answer 15

X(min-max) = X - X(min) / X(max) - X(min)

Question 16

What is one benefit and one deficit of min-max normalization

Answer 16

The y-axis is converted to 0 -> 1, no negatives, guarantees all features will have exact same scale.

Sensitive to outliers.

Question 17

How do you import MinMaxScaler from sklearn and use it to transform all the columns of the cars dataset (besides the last categorical column)?

Answer 17

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
cars_minmax = scaler.fit_transform(cars.iloc[:, :-1])
cars_minmax = pd_DataFrame(cars_minmax)

Question 18

How is the Z-Score transformation function defined?

Answer 18

X (z-score) = X - X(mean) / X (std)

Question 19

How do you calculate the standard deviation of a dataframe, df, column using numpy?

Answer 19

np.nanstd(df.column)

Question 20

How do you fit a standard normalization transformation to all columns of the cars dataframe except for the last one using sklearn.preprocessing?

Answer 20

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
cars_zscore = scaler.fit_transform(cars.iloc[:,:-1])
cars_zscore = pd.DataFrame(cars_zscore)
cars_zscore.columns  = cars.iloc[:, :-1].columns

Question 21

How do you achieve normality?

Answer 21

Achieve symmetry

Question 22

How do you achieve symmetry?

Answer 22

Reduce skewness

Question 23

How do you calculate the skewness of a distribution?

Answer 23

3(mean-median)/std

Question 24

How do you interpret skewness based on mean and median values?

Answer 24

mean > median | Right skew
mean < median | Left skew
mean = median | No skew (normal distribution)
(positive number : right)
(negative number: left)

Question 25

Do min-max or z-score standardization affect skewness?

Answer 25

No

Question 26

What are the three ways to reduce skewness?

Answer 26

1. Log Transformation
2. Square root transformation
3. Inverted Square Root Transformation

Question 27

How do you apply a log transformation to a numerical column?

Answer 27

df.column.apply(np.log)

Question 28

How do you apply a square root transformation to a numerical column?

Answer 28

df.column.apply(np.sqrt)

Question 29

How do you apply an inverted square root transformation to a numerical column?

Answer 29

df.column.apply(lambda x: np.reciprocal(np.sqrt(x)))

Question 30

How do you show a Q-Q plot for a data distribution using sipy.stats?

Answer 30

import scipy.stats as stats

stats.probplot(distribution_column. dist = “norm”, plot = plt)