Chp 5 Decision Tree

Question 1

Greedy Strategy

Answer 1

Split the records based on an attribute test that optimizes certain criterion

Question 2

Binary Split

Answer 2

Divides values into two subsets

Question 3

Multi-way split

Answer 3

Use as many partitions as there are distinct values

Question 4

How to specify continuous attributes

Answer 4

sort attribute
create split positions, at halfway points
Determine gini index

Question 5

The lower the gini index

Answer 5

the better it is

Question 6

Gini Index

Question 7

Entropy

Answer 7

Amount of uncertainty involved in a distribution

Question 8

In decision tree algorithms, entropy measures

Answer 8

Purity

Question 9

Purity

Answer 9

The fraction of observations belonging to a particular class in a node

Question 10

Pure node

Answer 10

If all observations belong to the same class

Question 11

Information

Answer 11

Amount of certainty involved in a distribution

Question 12

We need to choose a split in a decision tree that maximizes

Answer 12

Information Gain

Question 13

Information Gain formula

Answer 13

Entropy before split - Entropy after split

Question 14

Steps to determine split?

Answer 14

Calculate entropy at root node
Calculate each information gain for each attribute split
Pick the attribute split that has the highest information gain

Question 15

Decision trees are non/parametric

Answer 15

Non parametric because you are not specifying any parameters

Question 16

Time complexity of building a decision tree

Answer 16

O(log n)

Question 17

Does multicollinearity affect decision tree accuracy?

Answer 17

No, just added extra height

Question 18

Finding optimal decision tree is

Answer 18

expensive, NP complete

Question 19

Decision tree can create

Answer 19

rectilinear boundaries

Question 20

Decision trees cannot create

Answer 20

non straight (only up or down) lines

Question 21

Model =

Answer 21

Algorithm + hypothesis

Question 22

4 steps of selecting a model

Answer 22

Prepare training data
Choose hypothesis set and algorithm
Tune algorithm
Train the model, fit the model to out of sample data (test set) and evaluate results

Question 23

Goal of model selection

Answer 23

Select the best model from training phase

Question 24

Two ways to evaluate models

Answer 24

Model Checking
Performance Estimation

Question 25

The best model is the one that gives you _ and _ well on testing set

Answer 25

Smallest prediction error
Generalizes

Question 26

Model Checking

Answer 26

Given a data set, divide it into training and testing dataset

Question 27

What happens if you randomly select test points that are not representative of the population in general?

Answer 27

Cross Validation

Question 28

Cross Validation approaches 3

Answer 28

Holdout
K-fold cross validation
Leave one out cross validation

Question 29

Cross validation

Answer 29

an approach to
systematically create and evaluate multiple models on
multiple subsets of the dataset

Question 30

Holdout method

Answer 30

Split 80% training data and 20% testing data randomly

Question 31

k-fold cross validation

Answer 31

Split data into k chunks, train on k-1 chunks and test on the kth chunk. Do this k times and calculate average error

Question 32

Leave one out cross-validation

Answer 32

Extreme version of k-fold where k=1 observation (n chunks)

Question 33

Use of kfold or LOOC resampling methods are more robust if

Answer 33

data is split into training, validation, and testing

Question 34

Typical application of holdout methods is to

Answer 34

determine a stopping point with respect to error, stop when test set error starts increasing (this is overfitting)

Question 35

why split data into three part?

Answer 35

If you model has certain hyper parameters, then you can adjust the hyperparameters on the validation dataset

Question 36

two ways to do performance evaluation of a model

Answer 36

Confusion Matrix
Receiver Operating Characteristics (ROC) curve

Question 37

Confusion Matrix

Answer 37

Provides numerous metrics computed from the matrix

Question 38

ROC curve

Answer 38

Characterize the trade off between positive hits and false alarms

Question 39

Can you minimize both FP and FN?

Answer 39

No

Question 40

What does the ROC plot?

Answer 40

The true positive rate against the false positive rate

Question 41

What does the middle line mean

Answer 41

a random guess

Question 42

AUC standards

Answer 42

.5-.6 fail
.6-.7 worthless
.7-.8 poor
.8-.9 good
>.9 excellent

Question 43

Overfitting

Answer 43

picking up nuances in training data, matches too much with training data

Question 44

Underfitting

Answer 44

model is too simple that it cant capture patterns

Question 45

Prepruning

Answer 45

halt growth of tree based on some constraint
good for shorter trees
dont know when to stop

Question 46

Post pruning

Answer 46

Grow tree to maximum size, then trim
Gives better results
Wastes computer cycles

Question 47

How to prune

Answer 47

Focus on complexity parameter (cp), keep splitting until cp reaches a certain value

Question 48

How to find cp

Answer 48

Use cross validation error and see where it starts to rise again after decreasing, use cp at this value

Question 49

What does pruning do

Answer 49

Gives tree that is more generalized

Question 50

Mostly all datasets have what type of class distribution

Answer 50

imbalanced

Question 51

Cost sensitive learning

Answer 51

Penalizes the model when it commits a false negative error

Question 51

Mitigation balance techniques 3

Answer 51

Cost sensitive learning
Sampling techniques
Synthetic data

Question 52

Synthetic data

Answer 52

May be generated, if possible, to ensure that the class distribution is equivalent

Question 53

Sampling techniques

Answer 53

modify the class distribution such that the rare class is well represented in the training set

Question 54

Undersampling + con

Answer 54

Gathers less of the majority class observations for training
Useful observations may not be part of the sample

Question 55

Oversampling + con

Answer 55

gathers more of the minority class observations for training
If training data is noise, oversampling may amplify the noise