Lecture 9 - Decision Trees

Question 1

Decision Tree

Answer 1

• Data-driven method
• Popular classification technique

Reasons

• Performs well across a wide range of situations
• Does not require much effort from the analyst
• Easy understandable by the consumers
  
  • At least when the trees are not too large
• Can be used for both:
  
  • Classification, called classification trees
  • Prediction, called regression trees

Question 2

Example

Answer 2

Question 3

Nodes

Answer 3

• Conditions in the nodes give the splitting value on a predictor
• The number inside the node gives the records after the split
• The bracket provides the number of records per class: [not acceptor, acceptor]
• The leaf nodes, named terminals, are marked with color to indicate a non-acceptor (orange) or acceptor (blue)

Question 4

Trees are easy translated into a set of rules

Answer 4

Question 5

Induction (with a Greedy Strategy)

Answer 5

• Tree is constructed in a top-down recursive divide-and-conquer manner
• St start, all the training instances are at the root
• Instances, i.e., from the training set, are then partitioned recursively based on selected attributes

Question 6

Issues with Induction (with a Greedy Strategy)

Answer 6

• Determine how to split the records
  
  • How to specify the attribute test condition?
  • How to determine the best split?
  • Determine when to stop splitting

Specifying Test Condition

• Depends on attribute type:
  
  • Nominal
  • Ordinal
  • Continuous
• Depends on number of ways to split:
• Binary split, i.e., 2-way
• Multi-way split

Question 7

Splitting based on nominal attributes

Answer 7

Question 8

Splitting based on Continuous Attributes

Answer 8

Discretization vs binary

Question 9

Determining the Best Split

Answer 9

Question 10

Information gain

Answer 10

• Used to determine which feature/attribute provide the maximum information about a class
• Split records based on an attribute test optimising certain criterion
• Need a measure of node impurity, e.g., Gini Index, Entropy, etc.

Question 11

Information gain (visual)

Answer 11

Question 12

Gini Index

Answer 12

Question 13

Entropy measure

Answer 13

Question 14

Combined impurity

Answer 14

Question 15

Categorical Attributes

Answer 15

Question 16

Stopping Criteria for Tree Induction

Answer 16

• Stop expanding a node when all the records belong to the same class
• Stop expanding a node when all the records have similar attribute values
• Early termination (to be discussed later)

Question 17

How to Address Overfitting

Answer 17

Pre-Pruning

• Stop the algorithm before it becomes a fully-grown tree
• Typical stopping conditions for a node:
  
  • Stop if all instances belong to the same class
  • Stop if all attribute values are the same
• More restrictive conditions:
  
  • Stop if number of instances is less than some user-specified threshold
  • Stop if expanding the current node does not improve impurity measures, e.g., Gini or information gain

Question 18

How to Address Overfitting

Answer 18

Post-pruning

• Grow decision tree to its entirety
• Trim the nod es of the decision tree in a bottom-up fashion
• If generalisation error improves after trimming, replace sub-tree by a leaf node
• Class label of leaf node is determined from majority class of instances in the sub-tree

Question 19

Pros and cons of decision trees

Answer 19

Advantages:

• Easy to understand (domain experts love them)
• Easy to generate rules

Disadvantages

• May suffer from overfitting
• Classifies by rectangular partitioning (so does not handle correlated features very well)
• Can be quite large - pruning is necessary
• Does not handle streaming data easily
  
  • … but a few successful ideas/techniques exist