Module 3 Flashcards
Hyperparameter
Model parameters that are chosen before the training
Hyperparameter tuning
- split dataset into training/validation/test
- split between 60/20/20 and 80/10/10
- try different hyperparameters
- select best according to the validation dataset accuracy
Cross-validation
- divide dataset into k equal folds
- k-1 folds for training+validation
- 1 for testing
- iterate k times
- testing on a different portion of the data
- performance on all k held-out test sets can be averaged
Cross-validation parameter tuning
Each iteration:
- 1 fold for testing
- 1 fold for validation
- k-2 folds for training
Cross-validation in production
- can use all the available data for training the model ✅
- don’t have a way of estimating the performance of the final trained model any more ❌
Confusion matrix: accuracy
TP + TN / (TP + TN + FP + FN)
Confusion matrix: precision
TP / (TP + FP)
Recall
TP + / (TP + FN)
Macro-averaged recall
Average of recall for each class
F-measure
F1 = 2 . precision . recall / (precision + recall)
Micro-averaging
Calculate the average of each metric (e.g. TP, FP, TN, FN)
Mean squares error
1/N Σ (yi - ŷi)**2
Imbalanced test set: solutions
- downsample the majority class
- up sample the minority class
Overfitting
- good performance on training data
- poor generalisation on other data
Underfitting
- poor performance on training data
- poor generalisation to other data
