Week 9 - Model ensembles

Question 1

What are the three types of models based on trees

Answer 1

1. Decision trees
2. Random forest
3. Gradient boosting

Question 2

What are model ensembles

Answer 2

The wisdom of crowds for machines. Combinations of models are known as model ensembles

Question 3

if p>1/2 this implies

Answer 3

weak learnability

Question 4

What 2 things do ensemble methods have in common

Answer 4

• They construct multiple, diverse predictive models from adapted versions of the training data.
• They combine the predictions of these models in some way, often by simple averaging or voting

Question 5

What is bootstrapping

Answer 5

Bootstrapping is any test of metric that uses random sampling with replacement, and falls under the broader class of re-sampling methods. Boostrapping assigns measures of accuracy to sample estimates.

Question 6

What is the 3 step process creating a bootstrap sample

Answer 6

1. Randomly sample one observation from the set
2. Write it down
3. Put it back in the set

Question 7

Bootstrapping contains duplicates for

Answer 7

Diversity

Question 8

What is bootstrap AGgregating (Bagging) ensembling method

Answer 8

• create multiple random samples from the original data using bootstrapping
• Creating different models (learners) from different random samples of the original data

Question 9

What is subspace sampling?

Answer 9

Encouraging the diversity in ensembles by building models from a different random subset of features instead of all features

Question 10

What is random forests

Answer 10

Random forests is an ensemble learning approach for various tasks (regression, classification) that create various decision trees at training time and output the class that is the mode of the classes (classification) or mean/average prediction (regression) of the individial trees.

Question 11

What is boosting

Answer 11

Boosting is an ensemble method for reducing the error in supervised learning by converting weak learners into strong ones. The goal is that each learner can do a bit better than the previous one by iteratively considering the error rate and try to improve by focusing on the points they did not perform very well on.

Question 12

How can we improvel learning with boosting?

Answer 12

By giving the misclassified instances a higher weight, and modifying the classifier to take these weights into account

Question 13

How can we assign weights (boosting)

Answer 13

We want to assign half of the weights to misclassified items and the other half to the correctly classified items
* Every item 1/|D|
* Weight of all misclassified items: e
* Weight of all correctly classified items: 1-e

Question 13

what is epsilon in boosting

Answer 13

the error rate. FP+FN/Total

Question 14

What are 3 sources of misclassification erros

Answer 14

1. Unavoidable bias
2. Bias due to low expressiveness of models
3. High variance

Question 15

What is unavoidable bias

Answer 15

if instances from different classes are describd by the same feature vectors

Question 16

What is bias due to low expressiveness of models

Answer 16

if the data is not linearly seperable then even the best liear classifier will make mistakes.

Question 17

What is high variance

Answer 17

A model has high variance if its decision boundary is highly dependent on the training data.

Question 18

What is bagging predominantly for?

Answer 18

Bagging is predominantly a variance-reduction technique. It is often used in combination with high-variance models such as tree models.

Question 19

What is boosting predominantly for?

Answer 19

Boosting is primarily a bias reduction tenchnique. It is typically used with high-bias models such as linear classifiers or univariate decision trees.

Question 20

What is a meta-model

Answer 20

a model that best combinees the predictions of base models

Question 21

What is stacking

Answer 21

Stacking involves training a learning algorithm to combine the predictions of several other learning algorithms.
* several models are trained using the available data
* a learning algorithm is trained to make a final prediction using the predictions of the other algorithms.

Question 22

Name 2 types of experiments we would conduct on ml models

Answer 22

1. on one specific dataset
2. on a varied set of datasets

Question 23

what is cross-validation

Answer 23

Randomy partition the data in k folds, set one fold aside for testing, train a model on the remaining k-1 folds and evaluate it on the test fold. This process is repeated k times until each fold has been used for testing once.

Question 24

What does cross-validation accomplish

Answer 24

By averaging over training sets we get a sense of the variance of the learning algorith. Once we are satisfied with the performance of our learning algorith, we can run it over the entire data set to obtain a single model.

Question 25

what cross validation do we use if we have very few training instances?

Answer 25

leave out out cross-validation

Question 26

what is leave one out cross validation

Answer 26

alternatevly we can we k=n and train on all but one test instance, repeated n times. This means that in each single-instance ‘fold’ our accruacy estimate ir 0 or 1, but by averaging n of those we get an approximetely normal distribution by the central limit theorem.

Question 27

What is null hypothesis

Answer 27

is the hypothesis that there is no significant difference between specified distributions

Question 28

what is significance test

Answer 28

A test of significance is a formal procedure for comparing observed distibutions of data with a hypothesis

Question 29

what is a p-value

Answer 29

the probability of obtaining a measurement of a certain vlaue or higher given the null hypothesis

Question 30

what is a t-test

Answer 30

a t-test of a type of inferential statistic used to determine if there is a significant difference between the means of two groups, which may be relate in certain feaures.

Question 31

when conducting an experiment with 1 dataset

Answer 31

use paired t-test

Question 32

When comparing the perfroamance of a pair of algorithms over multiple dataset use

Answer 32

Wilcoxon’s signed-rank test

Question 33

What is Wilconxon’s signed-rank test

Answer 33

• the idea is to tank the performance differences in absolute value, from smallest to largest
• We then calculate the sum of ranks for positive and negative differences seperately and take the smaller of these sumbs as our test statistic
• Null hypothesis: two algorithms perform equally on multiple data sets

Question 34

what is a critical value in Wilcoxon’s singed-rank test

Answer 34

The critical value (the value of the test statistic at which the p-value equals alpha) can be found in a statistical table and used for rejecting the null hypothesis.

Question 35

How to compare multiple algorithms over multiple data sets

Answer 35

Freidman test

Question 36

What is the friedman test

Answer 36

• The idea is to rank the performance of all k algorithms per data set, from best performance to worst performance
• R_ij -> the rank of the j-th algorithm on the i-th data set

Question 37

What 3 quantities do we need to calculate in the Friedman test

Answer 37

1. The average rank
2. The sum of squared differences ( the spread between the rank centroids)
3. The sum of the squared differences ( the spread over all ranks)

Question 38

What is the friedman statistic?

Answer 38

The ration of the second to third quantity.

Question 39

What is a post-hoc test

Answer 39

• The idea is to calculate the critical difference
• The nemenyi test calculates the critical difference