Anomoly Detection

Question 1

What are anomalies/outliers?

Answer 1

The set of data points that are very different than the remainder of the data

Question 2

What is the task for looking for anomolies/outliers?

Answer 2

Find all the data points with anomaly scores greater than threshold that you have defined.

Question 3

What are some applications for Anomaly/Outlier detection?

Answer 3

Fraud detection

Question 4

Are outliers different from noise data?

Answer 4

Yes. Noise is random error.
Noise should be removed before outlier detection.
Outliers are interesting.

Question 5

What’s the difference between outlier detection vs novelty detection?

Answer 5

Novelty is eventually

Question 6

What is a challenge of anomaly detection?

Answer 6

Anomaly detection is unsupervised (like Clustering).

Question 7

How do you build an anomaly detection?

Answer 7

Build a profile of what is normal and then detect anything that is different

Question 8

What kind of outliers are there?

Answer 8

Global Outliers
Contextual Outliers
Collective Outliers

Question 9

What is a global outlier?

Answer 9

A point that significantly deviates from the rest of the data set.

Issue: You need a measurement of how you measure this

Question 10

What is a contextual outlier?

Answer 10

An outlier that deviates significantly based on selected context

E.g Is 40 degrees Celsius an outlier? In winter, yes. In summer, no.

Question 11

What are collective outliers?

Answer 11

Every object doesn’t look like an outlier but when you bring many objects together, it starts to look like an outlier.

Example: Sports/team: A good player Neymar is just like Messi or Ronaldo. But when you put them together with a good team they become an anomaly.

Question 12

What is a statistical schemes?

Answer 12

The objects are generated by a model.

Identify objects in low probability regions of the model as outliers.

Two types: Parametric/Non-parametric

Question 13

What is parametric model?

Answer 13

A model that describes the distribution of the data

If something in the model has low probability, then it is an outlier.

Find the mean and the standard deviation.
Check each the difference from the average. If it is greater than a threshold, then it is an anomaly.

Question 14

What is a limitation of a parametric scheme?

Answer 14

Not always a normal distribution

Can be problematic for high dimensional data

Question 15

What do you use to model a non-parametric scheme?

Answer 15

A histogram

Question 16

How do we interpret a histogram for determining anomaly detection?

Answer 16

The ‘long tail’ part of the histogram is considered the anomaly area of the model.

Question 17

What is a problem with using a histogram for analysis of anomaly detection?

Answer 17

How to set the number of buckets (x-axis) to effectively capture the data

Question 18

What is a false positive in anomaly detection?

Answer 18

An anomaly that is in fact that an anomaly. (Our histogram is too detailed).

Question 19

What are the two methods for detecting proximity based outliers?

Answer 19

1. Distance-based

2. Density-based

Question 20

What is distance based outlier?

Answer 20

An object is considered a distance based outlier if it’s neighbourhood doesn’t have enough other points.

Question 21

What is density based outlier?

Answer 21

An object is considered a density-based outlier if its density is relatively much lower than it’s neighbours

Question 22

What is LOF method for finding density based outlier?

Answer 22

General idea: For each point, calculate the density of it’s neighbourhood.

Compute: Local Outlier Factor: it’s the average of the ratio of density of the sample p and the density of it’s nearest neighbour

Outliers are the points with low LOF.

Question 23

How do we measure density?

Answer 23

Density = k / distance to the k-nearest neighbours, or compare with the set of N - nearest neighbours

Question 24

Can you get a different result for density/distance based outliers?

Answer 24

Yes

Question 25

If you have clustering, how do you determine if there are outliers?

Answer 25

It doesn’t belong to a cluster.
There is a large distance between an object and it’s cluster.
It belongs to a very small or sparse cluster

Question 26

What is Case 1: Far from closest cluster way of testing outliers?

Answer 26

Use k-means and build clusters, get an outlier (measure the distance to its closest centre. If it’s distance is higher than average then it is likely an outlier

Question 27

What is Case 2: Outliers in small clusters way of testing outliers?

Answer 27

Assign a cluster-based local outlier factor.

If p belongs to a large cluster: CBLOF = cluster size * similarity between P and Cluster

If p belongs to a small cluster: CBLOF = cluster size * similarity between p and the closest large cluster

LOW CBLOF scores are suspected outliers`

Question 28

What is a limitation of a cluster-based method?

Answer 28

High computational cost