Lecture 4: Decision trees & k-means Flashcards
What is logical inference?
Process of deriving new facts from a set of premises
What are the 3 types of logical inference?
1.Deduction
2.Abduction
3.Induction
What is deduction?
-Conclusion follows necessary from the premises
What is abduction?
-Conclusion is one hypothetical(most probable) explanation for the premises
What is induction?
-Conclusion about all members of a class from the examination of only a few member of the class.
What is inductive learning?
-Most work in ML
-Examples are given to train a system in a classification task
What are 3 examples of techniques in ML?
-Probabilistic methods
-Decision trees
-Neural networks
What are decision trees(3)?
-Simple, but very successful form of learning algo
-Very well-know algo is ID3 and its successor C4.5
What is the Ockham’s razor principle?
-Always favor the simplest answer that correctly fits the training data (the smallest tree on average)
What type of assumption is Ockham’s razor principle based on?
Inductive bias : making a choice beyond what the training instances contain
What is maximum information-gain and what uses it?
-Choose the attribute that has the largest information gain.
-ID3 uses it
What is essential information theory?
-Shannon in 1940s
-Notion of entropy(information content)
-Measure how predictable a random variable(RV) is if:
i. if you already have a good idea about the answer–> low entropy
ii. if you have no idea about the answer–> high entropy
What are the 2 things to watch out for in all types of learning?
-Noisy input
-Overfitting/underfitting the training data
What are 3 cases of noisy input?
-Two examples have the same feature-value pairs, but different outputs
-Some values of features are incorrect or missing
-Some relevant attributes are not taken into account in the data set
What are 3 cases of overfitting?
-Complicated boundaries overfit the data
-Large number of irrelevant features
-They do not generalize well to the new data
*Training error is high
*Testing error is high
