Lecture 7 - Outlier Detection, Feature Selection, Similar Items, Recommender Systems, Naive Bayes Classifiers, Class Imbalance

Question 1

What is an outlier?

Answer 1

An outlier is a data object that deviates significantly from normal objects as if it were generated by a different mechanism.

Question 2

What can cause outliers?

Answer 2

• Measurement errors
• Data entry errors
• Containment of data from different sources
• Rare events

Question 3

True or false: You should always try to remove outliers. It makes the machine learning algorithm better

Answer 3

FALSE: If the number of outliers are small, then it’s generally okay to remove them.
However, if the number is large, you have to think whether the outliers mean something, or whether it is ok to remove them

Question 4

What are some different methods to detect outliers?

Answer 4

• Model-based
• Graphical Approaches
• Cluster-based
• Distance-based
• Supervised-learning

Question 5

In model based outlier detection, in broad terms, how do we detect outliers?

Answer 5

1. We fit a probabilistic model
2. Outliers are cases with low probability

Example:

• Assume data follows normal distribution
• The z-score for 1D data is given by:

Question 6

What is the difference between a global and a local outlier?

Answer 6

A data point is a global outlier when it is out of the normal data range (i.e., most points are in a data range, and this data point is far out).

But, let’s say we have two clusters of points, and in between these clusters, there is a single standing point. We call this a local outlier because it is in the normal data range (given that to its right and left there are more data points), but cannot necessarily be assigned to any of the clusters

Question 7

Explain the approach of the Graphical Outlier Detection

Answer 7

We plot the data and look for weird points

We (Human) decides whether a data point is an outlier

Question 8

Can outliers be represented by groups?

Answer 8

Yes, they can. But remember, if the group has a relatively large number of data points, they might not be outliers, but they might be describing an “unusual” event that is worth to be captured in the data (so, do not remove them)

Question 9

What are some graphical representations that we can use to detect outliers by eye?

Answer 9

Boxplot - plot one variable at a time (and look at the outliers as single standing points, or even analyze the interquartile range)

Scatterplot - plot two variables at a time (is able to capture more complex patterns)

Question 10

How does cluster-based outlier detection work?

What are the main algorithms that can help in doing that?

Answer 10

Cluster data and then find points that do not belong to any of the clusters.

1. K-means: find points that are far away from any mean (but they have been categorized as a part of the cluster) or find clusters that have a small number of data points
2. Density-based clustering: outliers are the points that have not been assigned to any cluster
3. Hierarchical clustering: outliers take longer to be assigned to a group

Question 11

How does distance-based clustering outlier detection work? (KNN outlier detection)

Answer 11

For each data point, compute the average distance to its KNNs.
Sort the set of N average distances.
Choose the biggest values as outliers.

btw: KNN was proved to be the most efficient in detecting global outliers

Question 12

How does supervised outlier detection work?

Answer 12

(I think) you can get a training dataset that has a column saying whether x is an outlier or not.
And you can use it to detect further outliers.

Question 13

When you want to find outliers graphically, you can analyze the IQR (interquartile range) in a box plot.

What is special about this IQR when the dataset has outliers?

Answer 13

When a dataset has outliers, the interquartile range is often able to summarize the variability in the data.

IQR = Q3 - Q1

Question 14

What are the advantages and disadvantages to supervised outlier detection?

Answer 14

Advantage:
- Can find very complicated outlier patterns

Disadvantages:

• Is supervised, i.e. needs column labeled “outlier”
• Can not detect new “types” of outliers

Question 15

How can we define the process of “Feature Selection”?

Answer 15

Feature selection works by selecting features that are “relevant” for predicting the target variable (so variables that have a strong relationship with the target variable)

Question 16

Name the different approaches of feature selection?

Answer 16

Association approach
Regression weight approach
Search and Score methods
Forward Selection
Backward Selection
Recursive Feature Elimination

Question 17

Explain how the Association approach of feature selection works.

Answer 17

1. For each feature compute the correlation between the feature values and the target value y
2. Say that the feature is relevant if the correlation is above or below some threshold (0.9 and -0.9, for example)

Question 18

True or False. The Association approach in feature selection is basically a sequential “hypothesis testing” process of the correlation between the variables.

Answer 18

True

Question 19

What are some downsides of the Association approach in Feature Selection?

Answer 19

It can give bad results as it ignores variable interactions

“Taco Tuesday” and “Coke and Mentos” example

Question 20

Explain how the “Regression Weight” approach of Feature Selection works.

Answer 20

1. Fit regression weights based on all features (for example, with least-squares method)
2. Approve all the features whose weight is higher than some threshold

Question 21

Name one advantage and one disadvantage of using the Regression Weight approach in Feature Selection

Answer 21

Advantage: can solve the issue of variable interactions that is present in the Association approach

Disadvantage: has issues with collinearity

Question 22

Explain the process of the “Search and Score” method in Feature Selection.

Answer 22

1. Define a score function that measures the quality of a set of features
2. Search for the set with the best score

The sets are represented by all the possible combinations of features in the dataset (including the empty set - so no features)

Question 23

How do you choose the score function in the “Search and Score” method in Feature Selection?

Answer 23

Simon Sardorf write here please

Question 24

How does Forward Selection (in Feature Selection) work?

Answer 24

1. Compute Score with no feature
2. Start adding features (one at a time) and compute the score
3. Keep the feature with the best score
4. Add another feature (one by one add all the different features)
5. Compute Score
   and so on..
   STOP when no single variable addition improves the score

Question 25

What is the downside of Forward Selection, and why should we use it despite this downside?

Answer 25

Might not be able to capture the best variables

But it is cheaper, overfits less, and has fewer false positives

Question 26

How does Backward Selection differ from Forward Selection?

Answer 26

It starts with computing the score of a set of all features, and then takes out one variable at a time to see if the score improves or not (removes the variable that results in the highest score improvement)

Question 27

How is user-product matrix used in Recommender Systems?

Answer 27

Column - products
Rows - users

1 shows whether a user has purchased the specific product
0 shows the opposite

Question 28

What algorithm can be used to find similar products (in terms of the users that purchased them) in a user-product matrix?

Answer 28

KNN

Question 29

What is the “closest point” problem (exists in KNN, K-means, Density-based clustering, also Recommender Systems?

Answer 29

“closest point” problem: let’s say that there are new products that are much better (to recommend), but they are considered as outliers → because that they have fewer people who bought them until now

How does this relate to KNN, K-means, and Density-based clustering?

Question 30

What are the two types of Recommender Systems? What is the difference between them?

Answer 30

Content filtering: assumes access to side information about items (supervised learning)
e.g., Gmail “important messages””

Collaborative filtering: does not assume access to side information about items (unsupervised learning)
e.g., Netflix
if Alice likes A and B, and Bob likes A, then Bob is likely to like B as well

(does not really work for new users - cause we don’t know their preferences yet)

Question 31

What are the two types of Recommender Systems?

Answer 31

Content filtering and collaborative filtering

Question 32

What are the two methods used in Collaborative Filtering Recommender Systems? Explain them.

Answer 32

Neighborhood method: Find neighbors based on similarity of movie preferences and then recommend movies that those neighbors watched

Latent Factor method: Assume that both users and items live in a low dimensional space describing their properties and then recommend items based on their proximity to the user in the latent space

Question 33

True or False. Matrix Factorization is a way to define the model of Collaborative Filtering.

Answer 33

True

Question 34

Bayes Classifier is built on … that is based on …

Answer 34

…Bayes Theorem… …conditional probability…

Question 35

What kind of model is Bayes Classifier?

Answer 35

Bayes Classifier is a supervised probabilistic model that makes the most probable predictions for new examples

Question 36

What other model is used to solve the issue of Bayes Classifier assuming variable dependence on each other?

Answer 36

Naive Bayes classifier

that assumes conditional independence of input variables

Question 37

What question does Naive Bayes classifier answer?

Answer 37

What is the most probable classification of a new instance given the training data?

Question 38

Explain how Naive Bayes Classifier can be used in Spam Filtering.

Answer 38

Collect many emails, get users to label them (spam/not spam)

Extract features of emails (like bag of words) and create columns for each word

1 if the word is in email, 0 if it is not in email

Use Gaussian NB function in Python to fit the model (see notes in Notion for more details)

Question 39

Name two techniques that deal with oversampling and one technique that deals with undersampling

Answer 39

Oversampling: SMOTE, ADASYN
Undersampling: cluster

Question 40

What is the purpose of conducting over/undersampling?

Answer 40

To fix class imbalance. To reach a ratio of 1:1 in the classes that we use

Question 41

What is the requirement that we have to satisfy before oversampling (synthetically generate data) for the process to work well?

Answer 41

when you create synthetic data, the data should be having less noise, with fewer outliers and fewer missing values, so the model can create a good sense of missing data that it should add

Question 42

How does the SMOTE algorithm work?

Answer 42

1. Find KNN from each sample
2. Select samples randomly from a KNN
3. Find new samples = {original samples + difference * gap(0,1)}
4. Add new sample to minority

A new dataset is created

Question 43

What is the complexity of a Forward Selection algorithm?

Answer 43

O(dˆ2) - quadratic

FS starts by choosing one variable and testing the performance of the model with that one variable only. But in this process of choosing, it needs to compute a score for the model containing each variable at a time (d times - if there are d features). Let’s assume it chooses one variable.

Then it needs to choose another one. This means that it needs to test again by adding each variable to the model ((d-1) times - because it already chose one previously, it cannot choose the same one again). Let’s assume it chooses another variable. and so on

After this whole process, it means that, in the worst-case scenario, the model was tested

d + (d-1) + (d-2) + … + 2 + 1 times. This equals d(d + 1)/2. This equals (dˆ2 + d)/2.

This means that the complexity of the model is O(dˆ2)

Question 44

What is the complexity of a Search and Score algorithm?

Answer 44

O(2ˆd) - exponential

Because Search and Score analyzes all possible combinations of sets that can be formed from the variables

Let’s assume we have the following variables {a, b, c}
The algorithm is going to test the performance of the model with the following variables:
{ }, {a}, {b}, {c}, {a, b}, {a, c}, {b, c}, {a, b, c}
8 = 2ˆ3

2ˆn is the total number of subsets that you can build from the variable set containing n elements

Therefore, if we want to use Search and Score to select feature, the algorithm is going to test the model 2ˆn times

So, the complexity of Search and Score is O(2ˆn)

Question 45

What is the complexity of a Search and Score algorithm?

Answer 45

O(2ˆd) - exponential

Because Search and Score analyzes all possible combinations of sets that can be formed from the variables

Let’s assume we have the following variables {a, b, c}
The algorithm is going to test the performance of the model with the following variables:
{ }, {a}, {b}, {c}, {a, b}, {a, c}, {b, c}, {a, b, c}