Basic Data Preparation

Question 1

What is an outlier?

Answer 1

a data point that’s very different from the rest

Question 2

point outliers

Answer 2

values far from the rest of the data

Question 3

contextual outlier

Answer 3

value isn’t far from the rest overall,but is far from points nearby in time

Question 4

collective outlier

Answer 4

something is missing in a range of points, but can’t tell exactly where

Question 5

box and whisker plot

Answer 5

visual to find outliers in one dimension

Question 6

another approach to outlier detection

Answer 6

fit exponential smoothing model
-points with very large error might be an outlier

Question 7

What causes outliers?

Answer 7

-bad data: sensor failure, contaminated experiment, wrong data input
-or it could be real data

• need to investigate

Question 8

Dealing with outliers

Answer 8

bad data - omit data, imputation (add in a better value)
real data - outliers are expected in large data sets

ex: normally distributed 4% of data outside 2 standard deviations
-with 1m data points >2000 would be outside 3 sd

Question 9

Dangers of dealing with outliers

Answer 9

-removing real data outliers can be too optimistic

-ex:
time to transport perishable medicine from us to africa

-outlying data points- weather events or political issues… should these be included in your model?

Question 10

alternative way to deal with outliers

Answer 10

logistic regression model - estimate probability of outliers to happen under different conditions

second model - estimate length of delivery under normal conditions
-use data without outliers

Question 11

draw box and whisker plot and name the different parts of it