Chapter 5 - Overfitting and its Avoidance

Question 1

What is overfitting?

Answer 1

The tendency of data mining procedures to tailer models to the training data, at the expense of generalization to previously unseen data points.

Question 2

What is a table model and what is the problem with it?

Answer 2

A tabble model memorizes the training data and preforms NO GENERALIZATION. This is also the problem, because the model can only calculate the known data 100%, but does not have any predictive power.

Question 3

What is generalization?

Answer 3

This is a property of a model or modeling process, whereby the model applies to data that was not used to build the model.

Question 4

What is the fitting graph

Answer 4

This is an analytical tool to recognize overfitting. It shows the accuracy of a model as a function of complexity.

Question 5

What is holdout data?

Answer 5

Data not used in building the model, but for which we do know the actual value of the target variable.

Question 6

How does the error rate of holdout data react to an increase in data points?

Answer 6

The holdout set error never decreases, because there is never an overlap between training and holdout sets.

Question 7

What is the base rate?

Answer 7

The base rate is the error rate of a holdout set. This is calculated as e.g. the percentage of churn cases in a population.

Question 8

Why does performance degrade due to overfitting?

Answer 8

As models become more complex, it is allowed to pick up harmful spurious correlations

Question 9

What it the holdout evaluation method?

Answer 9

This is a way of solving overfitting by dividing the data in test data (holdout data) and training data.

Question 10

What are the cons a holdout set?

Answer 10

1. It is a single estimate and therefore we too much focussed on one dataset (we could have had luck with a good dataset).

Question 11

What is cross-validation?

Answer 11

Solution to holdout set problems. It includes the mean and variance of estimated performance. The variance is critical in assesing confidence in the perfoemance estimate.

Question 12

What are the pro’s of cross-validation?

Answer 12

1. Makes better use of a limited dataset

2. Computes estimates over all the data by performing multiple splits and swapping out samples for testing.

Question 13

How does cross validation work?

Answer 13

1. Split dataset into k partitions (k = 5 or 10)
2. Iterates training and testing data k times.
3. Each iteration: training data = (k-1)/k and test data = 1/k
4. Calculate the average accuracy

Question 14

Doe you trust performance done on a training set and why?

Answer 14

No, due to the high chances of overfitting. Possible solutions are: Cross-validation and holdout methods

Question 15

Explain the learing curve.

Answer 15

This is a plot of generalization performance against the amount of training data. They are steep initially and flattens with bigger datasets due to a decrease in marginal advantages.

Question 16

What is the difference between learning curves and fitting graphs?

Answer 16

A learning curve shows the generalization performance ONLY ON TESTING DATA. (plotted against THE AMOUNT OF TRAINING DATA)
A fitting graph shows the generalization performance on BOTH PERFORMANCE AND TESTING DATA. (plotted against COMPLEXITY)

fitting graphs are generally shown for fixed amount of training data.

Question 17

Compare classification trees and linear logistic regression

Answer 17

Classification tress:
* Will overfit more for smaller datasets.
* Can work better for large datasets.
Logistic regression:
* Usually leads to better performance on smaller datasets

Question 18

What is the deal w/ tree induction

Answer 18

Main problems:

1. Tree will keep growing to fit the training data -> creates pure leave nodes.
2. Overly complex trees that overfit data.

Solutions:

1. Stop growing the tree before it gets too complex
2. Grow tree until it is too large and then “prune” back to reduce the size and complexity.

Methods for solution:

1. Limit tree size to specify a minimum number of instances that must be present in a leaf.
2. “Prune” an overly large tree (cut off leaves and branhes, and replace with leaves).
   * By