Paper 5 LearningCurves Flashcards
Why learning curves?
Data acquisition (how many more labels reasonable)
Early stopping (of training)
Early Discarding (in model selection)
Three criteria of the Learning Curves framework
Type of decision situation
The type of question being answered
The data resources that are used
Iteration based vs observation based
Iteration based: fixed dataset, learning over time
Observation based: varying dataset, learning over samples
Utility curve
The return on investment estimate for investing more resources
Well behaved learning curves
Convexity (systematix improvement)
Monotonixity (improvements aren’t lost)
Modelling a Learning Curve
Derive model of the true learning curve from the empirical one
Vertical Model Selection
Allows for an evolving set of learners, evaluating them one after another, and growing learning curves iteratively.
Horizontal Model Selection
Involves iteratively growing empirical learning curves for a fixed set or subsets of learning algorithms.
Diagonal Model Selection
Similar to vertical, but it allows continuing the evaluation of a candidate at a later point, interleaving evaluations of different candidates.
