Lecture 4 - Tree models

Question 1

What are the three models based on trees

Answer 1

1. decision trees
2. random forest
3. gradient boosting

Question 2

What are the three parts of a decision tree

Answer 2

1. root node
2. internal/decision nodes
3. leaf nodes

Question 3

what is the max depth and max number of leaves of a decision tree with d number of binary features?

Answer 3

d+1

Question 4

how many leafs does a decision tree have with d number of binary features

Answer 4

2^d

Question 5

Decision tree

The set of literals at a node is called a ___

Answer 5

split

Question 6

each leaf of the tree represents a ___ ___, which is a conjuction of literals encounteresd on the path from the root of the tree to the leaf

Answer 6

logical expression

Question 7

Give 3 decision tree algrithms

Answer 7

1. ID3
2. C4.5
3. CART

Question 8

Growing a tree is recusive
true/false

Answer 8

true

Question 9

how to decide the best split

Answer 9

to assign the majority class label to leaves, we look for a clear split between two classes -> Purity of the children of the split

Question 10

Give 3 impurity metrics

Answer 10

1. Minority class
2. Entropy
3. Gini index

Question 11

Both the entropy and the Gini index are smooth and concave upper bounds of the training error. These properties can be advantegeous in some situations

Answer 11

True

Question 12

What is entropy

Answer 12

Entropy is the expected average level of information, surprise, or uncertainty inherent in all the n possible outcomes of an event

Question 13

Entropy for a binary classification tasks:

Answer 13

H(p,1-p) = -p log_2 p-(1-p)log_2 (1-p)

Question 14

How do we assess if a split is useful at all?

Answer 14

In assessing the quality of a feature for splitting a parent node D into leaves, it is customary to look at the purity gain. Imp(D)-Imp(D1..Dj)
Purity gain = original entropy - entropy after splitting

Question 15

Finding the best split for a decision tree is recursive?

Answer 15

False

Question 16

Ho

How to prevent overfitting in decision trees. Give 2

Answer 16

• Limit the number of iterations of the algorithm leading to a tree with a bounded number of nodes.
• Prune the tree after it is built by removing weak branches.

Question 17

How to do Reduced Error Pruning

Answer 17

1. Starting at the leaves, each node is replaced with the majority class
2. If the predictino accuracy is not affected then the change is kept.
3. Keep a validatin set, see pruned tree performance on the validation set. Of course the pruning will not improve accuracy on the training set.

Question 18

What are the two sources of imballance

Answer 18

• Asymmetric class distribution
• Asymmetric mis-classification cost

Question 19

What does adding more samples accomplish?

Answer 19

Adds data for training and increases the training time
It is possible that it does not make any difference

Question 20

what is sqrt(gini)

Answer 20

Sqrt(gini) is designed to minimise relative impurity and this is insensitive to changes in class distribution, whereas Gini emphasises children covering more examples

Question 21

Entropy and Gini index are sensitive to fluctuations in the class distrubutions, sqrt(gini) isn’t. We want distribution-insensitive impurities.

Answer 21

yeah true

Question 22

In regression trees we can replace imp with ____

Answer 22

variance

Question 23

What is weighted variance

Answer 23

If a split partions the set of target values Y into mutually exclusive sets {y1,…yj}

Question 24

The variance of boolean variable with success probability p is p(1-p), which is half of the gini index. So we could interpret the goal of tree learning as minimising the calss variance in the leaves.

Answer 24

true

Question 25

In regression trees out goal is

Answer 25

to find a split that minimises the weighted average of the variance of the child nodes.