Measures

Question 1

Brier Score

Answer 1

• measures accuracy of prediction models
• mean square difference between predicted probability and actual outcomes
• 0 indicates perfect predictions

Question 2

ECE

Answer 2

Expected Calibration Error
* measures how well model’s predicted probabilities reflect the true likelihood
* partions into bins then weighted average across all bins
* low ECE is optimal

Question 3

MAE

Answer 3

Mean Absolute Error
* measure of accuracy of predictions
* measure absolute differences between predicted values and actual values
* low MAE - better predictions

Question 4

RMSE

Answer 4

Root Mean Square Error
* measure accuracy of model
* calculates square root of average square differences between predicted and actual values
* low RMSE - better predictions

Question 5

out-of-sample R^2

Answer 5

• compare predictive performance of two models by assessing how much one model improves upon the other in terms of MSE
• > 0 - model 1 better
• < 0 - model 2 better
• = 0 - same MSE