Data wrangling

Question 1

Replace missing values with naN

Answer 1

# replace "?" to NaN
df.replace("?", np.nan, inplace = True)

.replace(A, B, inplace = True)

Question 2

Identify missing data

Answer 2

The missing values are converted to default. We use the following functions to identify these missing values. There are two methods to detect missing data:

.isnull()
.notnull()

missing_data = df.isnull()

The output is a boolean value indicating whether the value that is passed into the argument is in fact missing data.

Question 3

Count missing values in each column

Answer 3

for column in missing_data.columns.values.tolist():
print(column)
print (missing_data[column].value_counts())
print(“”)

Question 4

See which value is most common

Answer 4

df[‘num-of-doors’].value_counts().idxmax()

Question 5

Drop rows

Answer 5

# simply drop whole row with NaN in "price" column
df.dropna(subset=["price"], axis=0, inplace=True)

axis = 0 drops the entire row 
axis = 1 drops the entire column

Question 6

Reset index

Answer 6

# reset index, because we droped two rows
df.reset_index(drop=True, inplace=True)

Question 7

List data types for each column

Answer 7

df.dtypes

Question 8

Convert data types to proper format

use .dtypes() to identify data types
use .astype() to convert data type

Answer 8

df[[“bore”, “stroke”]] = df[[“bore”, “stroke”]].astype(“float”)

df[[“normalized-losses”]] = df[[“normalized-losses”]].astype(“int”)

df[[“price”]] = df[[“price”]].astype(“float”)

df[[“peak-rpm”]] = df[[“peak-rpm”]].astype(“float”)

Question 9

Rename a column

Answer 9

rename column name from “highway-mpg” to “highway-L/100km”

df.rename(columns={‘“highway-mpg”’:’highway-L/100km’}, inplace=True)

Question 10

Design the bins

Answer 10

bins = np.linspace(min(df[“horsepower”]), max(df[“horsepower”]), 4)
bins

Question 11

Apply the cut function to a certain column

Answer 11

df[‘horsepower-binned’] = pd.cut(df[‘horsepower’], bins, labels=group_names, include_lowest=True )
df[[‘horsepower’,’horsepower-binned’]].head(20)

Question 12

Assign dummy variables to each column

Answer 12

dummy_variable_1 = pd.get_dummies(df[“fuel-type”])

dummy_variable_1.head()

Question 13

Change column names

Answer 13

dummy_variable_1.rename(columns={‘gas’:’fuel-type-gas’, ‘diesel’:’fuel-type-diesel’}, inplace=True)
dummy_variable_1.head()

Question 14

merge the data frams

Answer 14

# merge data frame "df" and "dummy_variable_1" 
df = pd.concat([df, dummy_variable_1], axis=1)

# drop original column "fuel-type" from "df"
df.drop("fuel-type", axis = 1, inplace=True)

Question 15

Methods of normalizing data

Answer 15

(1) simple feature scaling:
Xnew = X old/Xmax
(2) Min-Max:
Xnew = (Xold - Xmin) / (Xmax - Xmin) 
(3) Z-score:
Xnew = (Xold - mean)/SD

Question 16

Simple feature scaling code

Answer 16

df [“length”] = df [ “length” ] / df [“length”].max()

Question 17

Min-max in Python

Answer 17

df [“length”] = (df [“length”] - df [“length”].min())/ (df[“length”].max() - df[“length”].min())

Question 18

Z score

Answer 18

df[“length”] = (df[“length”]-df[“length”].mean())/df[“length”].std()