Cleaning Data

Question 1

What is data cleansing?

Answer 1

Removing/Fixing problem data

Question 2

What are some example data problems?

Answer 2

Duplicates
Empty rows
Abbreviations
Scale inconsistency
Typos
Missing values
Trailing spaces
Incomplete cells

Question 3

If there is ______ data in the data set, we can fix, delete, etc as appropriate

Answer 3

invalid

Question 4

An ______ is a data point that’s far away from others

Answer 4

outlier

Question 5

Outliers can occur in ______data and should not immediately be thrown away

Answer 5

valid

Question 6

How can we identify outliers?

Answer 6

Look at how for a point is from the mea, often how many standard deviations away from the mean

Question 7

What’s the first thing to do when trying to find outliers?

Answer 7

Plot the data and look for problems by eye

Question 8

What are 4 things we can do with outliers?

Answer 8

Leave it
Remove it
Remove the value and treat is as a missing value
Impute it

Question 9

What is imputation?

Answer 9

Replacing values with a calculated plausible value

Question 10

What are some drawbacks of imputation?

Answer 10

Can introduce bias
Introduce fake trends
Wash out real trends

Question 11

What are common ways to impute?

Answer 11

Average nearby values
Average of known data
Linear regression of nearby values

Question 12

What is entity resolution or record linkage?

Answer 12

The process of finding multiple values that actually refer to the same entity