classification (decision trees)

Question 1

What is the goal of decision trees?

Answer 1

To create a model that predicts the value of a target variable by learning simple decision rules inferred from features

Question 2

What are decision trees?

Answer 2

A non-parametric supervised learning method used for classification

Question 3

What are benefits of decision trees?

Answer 3

• don’t need as much data and gives rules that it learned
• no underlying assumptions
• handles multidimensional data
• achieves good accuracy

Question 4

What is the splitting criterion?

Answer 4

Tells us which attribute to test at node N by determining the “best” way to separate or partition tuples into individual classes

Question 5

What does it mean for a partition to be pure?

Answer 5

If all tuples belong to the same class

Question 6

What does information gain do?

Answer 6

Minimizes the info needed to classify tuples in resulting partitions and reflects least randomness or “impurity”

Question 7

(T/F): Info gain guarantees that a simple tree is found

Answer 7

True

Question 8

What does Gain(A) tell us?

Answer 8

Tells us how much would be gained by branching on A. It is the difference between original info required and new info required.

Question 9

How can you determine the “best” split point for A if it is CONTINUOUS-VALUED?

Answer 9

• sort of values of A in increasing order
• consider midpoint as split point ( ( ai + ai + 1 ) / 2 )

Question 10

(T/F): For info gain, pick the feature and split that MAXIMIZES weighted entropy

Answer 10

False; pick the feature and split that MINIMIZES weighted entropy

Question 11

What kind of DT is info gain used in?

Answer 11

ID3

Question 12

What kind of DT is the Gini index used in?

Answer 12

CART

Question 13

What is Gini Index?

Answer 13

An alternative measure of impurity of class labels among a particular set of tuples

Question 14

How many possible ways are there to form 2 partitions of the data based on a binary split on A?

Answer 14

2^v - 2 ways

Question 15

(T/F): We want to MINIMIZE the gini index

Answer 15

True

Question 16

How do you pick the splitting subset for DISCRETE-VALUED attribute using Gini index?

Answer 16

Subset that gives minimum Gini index for that attribute

Question 17

How do you pick the splitting subset for CONTINUOUS-VALUED attribute using Gini index?

Answer 17

• Consider each possible split-point
• midpoint bw each pair of sorted adjacent values is taken as a possible split point

Question 18

What is the range of the Gini index for a BINARY label?

Answer 18

[0, 0.50]

Question 19

What does maximum purity mean?

Answer 19

Every instance has the same label (in the binary case)

Question 20

What does maximum impurity mean?

Answer 20

50% of data are one label and 50% are another

Question 21

Explain the bias of info gain.

Answer 21

Biased towards multivalued attributes

Question 22

Explain the bias of the gain ratio.

Answer 22

adjusts for bias, but prefers unbalanced splits in which one partition is much smaller than others

Question 23

Explain the bias of Gini index.

Answer 23

biased towards multivalued attributes and has difficulty when the number of classes is large. It also favors tests that result in equal size partitions and purity in both partitions.

Question 24

What are the 3 partitioning scenarios?

Answer 24

1) A is discrete-valued
2) A is continuous-valued
3) A is discrete-valued and a binary tree

Question 25

Why is pruning performed?

Answer 25

To address overfitting

Question 26

What is pre-pruning?

Answer 26

“Pruned” by halting its construction early

Question 27

What are 2 examples of pre-pruning?

Answer 27

• max depth
• min leaf size

Question 28

What does controlling for min leaf size do?

Answer 28

The higher the leaf size, the less overfit it will get because it won’t create rules past a certain threshold. This means it could miss out on more detailed rules that capture something more generalizable.

Question 29

What is post-pruning?

Answer 29

Removes rules after decision tree algorithm has completed.

Question 30

Explain the cost-complexity ratio

Answer 30

This refers to post-pruning where we are trying to minimize an increase in error from pruning while also minimizing the number of rules used (less rules, less complex)