Lecture 12 - Decision Tree Induction Part 2

Question 1

What is the pseudocode for decision tree induction?

Answer 1

FUNCTION buildDecTree(examples,atts)

Create node N if necessary; //starting as a node, ending as a tree

IF examples are all in same class THEN RETURN N labelled with that class;

IF atts is empty THEN RETURN N labelled with modal example class;

bestAtt = chooseBestAtt(examples,atts);

label N with bestAtt;

FOR each value a i of bestAtt //each branch from node N

s i = subset of examples with bestAtt = a i;

IF s i is not empty THEN

newAtts = atts – bestAtt;

subtree = buildDecTree(s i,newAtts); //recursive

attach subtree as child of N;

ELSE

Create leaf node L;

Label L with modal example class;

attach L as child of N;

RETURN N;

Question 2

How is the best attribute usually chosen for decision tree induction?

Answer 2

Information Gain

Question 3

Explain the pseudocode for decision induction

Answer 3

Take the training set

Work out information gain for each attribute against the target attribute

Highest info gain is the best attribute to sort by

Examine the subtrees from separating by that attribute

If all the values of the target attribute are the same in the subtree, that subtree is replaced by a leaf with that value

Otherwise run the whole procedure again, unless there are no more attributes to sort by in which case choose the most frequently occuring value of the target attribute

Question 4

What are 3 issues with decision tree induction?

Answer 4

1) Inconsistent data
2) Numeric Attributes
3) Overfitting

Question 5

Why is inconsistent data a problem in decision tree induction?

How can it be solved?

Answer 5

May often have no more attributes available to generate subtree with

Easiest method to solve this is use modal value

Question 6

Why are numeric attribute values a problem in decision tree induction?

How can the issue be alleviated?

Answer 6

With numeric attribute values there is a very large number of possible values - making massive trees

Easy solution is to divide into ranges, e.g. 1-5, 6-10, 11-15 so reducing the number of possible values

Question 7

What is overfitting?

Answer 7

When a model starts to model random noise on top of the real data.

Question 8

What is overfitting specifically in relation to decision induction, and why does it occur?

How can it be alleviated?

Answer 8

When the decision tree models the sample set rather than the whole population and so takes into account peculiarities of the training set which might not be true of the whole population.

Can be alleviated by:

Pre pruning

Post Pruning

Increasing number of high quality samples