Chapter 28 How to Calibrate Predicted Probabilities

Question 1

Predicted probabilities that match the expected distribution of probabilities for each class are referred to as ____.

P 286

Answer 1

calibrated

we desire that the estimated class probabilities are reflective of the true underlying probability of the sample (Me: Instance/row). That is, the predicted class probability (or probability-like value) needs to be well-calibrated. To be well-calibrated, the probabilities must effectively reflect the true likelihood of the event of interest.

Question 2

Although a model may be able to predict probabilities, the distribution and behavior of the probabilities may not match the expected distribution of observed probabilities in the training data (it’s not calibrated). Which type of models encounter this issue more?

P 287

Answer 2

This is especially common with complex nonlinear machine learning algorithms that do not directly make probabilistic predictions and instead use approximations.

Question 3

There are two steps for calibrating probabilities; What are they?

P 287

Answer 3

• Diagnosing the calibration of predicted probabilities (Reliability Diagrams aka. Calibration Curves)
• The calibration process itself.

Question 4

How is a reliability diagram (aka. Calibration curve) created?

P 287

Answer 4

A reliability diagram is a line plot of the relative frequency of what was observed (y-axis) versus the predicted probability frequency (x-axis).
* Specifically, the predicted probabilities are divided up into a fixed number of buckets along the x-axis. (strategy: uniform of calibration_curve sklearn)
* y-axis=The proportion of samples whose class is the positive class, in each bin (fraction of positives).
* x-axis=The mean predicted probability in each bin.
* The results are then plotted as a line plot.

Calibration_curve Doc

Question 5

The better calibrated or more reliable a forecast, the closer the points will appear along the main diagonal from the bottom left to the top right of the reliability diagram. The position of the points or the curve relative to the diagonal can help to interpret the probabilities; For example:
ˆ Below the diagonal: The model has ____ (over-forecast/ under-forecast); the probabilities are too ____(small/large).
ˆ Above the diagonal: The model has ____ (over-forecast/ under-forecast); the probabilities are too ____(small/large).

P 288

Answer 5

over-forecast; large. under-forecast; small.

Question 6

The predictions made by a predictive model can be calibrated. Calibrated predictions always result in an improved calibration on a reliability diagram. True/False

P 288

Answer 6

False, the predictions made by a predictive model can be calibrated. Calibrated predictions may (or may not)result in an improved calibration on a reliability diagram.

Question 7

Some algorithms are fit in such a way that their predicted probabilities are already calibrated. ____ is one such example. Other algorithms do not directly produce predictions of probabilities, and instead a prediction of probabilities must be approximated. Some examples include ____, ____, and ____. The predicted probabilities from these methods will likely be uncalibrated and may benefit from being modified via calibration.

P 288

Answer 7

logistic regression, neural networks, support vector machines, decision trees

Note, and this is really important: better calibrated probabilities may or may not lead to better class-based or probability-based predictions. It really depends on the specific metric used to evaluate predictions. In fact, some empirical results suggest that the algorithms that can benefit the more from calibrating predicted probabilities include SVMs, bagged/boosted decision trees, and random forests.

Question 8

Calibration of prediction probabilities is a rescaling operation that is applied after ____.

P 288

Answer 8

The predictions have been made by a predictive model

Question 9

There are two popular approaches to calibrating probabilities; What are they called and when are they used?

P 288

Answer 9

they are the Platt Scaling and Isotonic Regression. Platt Scaling is simpler and is suitable for reliability diagrams with the S-shape. Isotonic Regression is more complex, requires a lot more data (otherwise it may overfit), but can support reliability diagrams with different shapes (is nonparametric).

Question 10

You can diagnose the calibration of a classifier by creating a reliability diagram of the actual probabilities versus the predicted probabilities on a test set. In scikit-learn, this is called a calibration curve. This can be implemented by first calculating the ____ function. A classifier can be calibrated in scikit-learn using the ____ class.

P 289

Answer 10

calibration_curve(), CalibratedClassifierCV