Choosing A Metric

Question 1

Key points in choosing a metric

Answer 1

1. You usually care about lots of metrics
2. Ml works best with 1 number to optimize
3. Pick a formule for combining metrics
4. It can and will change

Important to say - in the development you car about one metric, once you have prediction look back at the big picture and see if this is working well

Question 2

How to combine metrics

Answer 2

1. Simple average/ weighted average (precision + recall / 2)….
2. Threshold n-1 metrics, evaluate the nth. Used much more in real life.
   Meaning - choose the minimum of the rest of the important metrics and than from there work on one. For example if it’s an edge device model - make sure it’s small enough, then lower the recall… etc

Domain specific metrics ex. mAP (average precision or area under the precision recall curve)

Question 3

Thresholding metrics

Answer 3

Choosing which metrics to threshold - domain judgment (which metrics can you engineer around)
Which metrics are least sensitive to model choice
Which metrics are closest to the desirable value

Choosing metric value - domain judgment (what is achievable? What is acceptable downstream?)
How well does the baseline model do?
How important is this metric right now?