Cost Functions, Performance Measures and Optimization

Question 1

What is the Cost Function?

Answer 1

The cost function is also called the loss function.

It is the difference between the predicted values and the ground truth, y.

Question 2

What is the cost function used for regression?

Answer 2

We call it RMSE. Root-Mean-Square-Error

Question 3

What is the cost function that we use to classify?

Answer 3

We have used a modified RMSE that is modified so that it returns either 0 or 1 by multiplying by the ground truth

Question 4

What is regularization?

Answer 4

Regularization is something we use to prevent overfitting in our model.

We add some information to the cost function to change the algorithm

Question 5

What is overfitting?

Answer 5

Overfitting is when the learning rate is too high so that the algorithm will over adjust when near the best fitting vector.

It could also describe when the model has been trained with too much training sets so that it learns the noise and errors of the data as features.

Question 6

With classifiers, usually, how many nodes should there be in each layer as we go towards the output layer?

Answer 6

There should be fewer and fewer nodes.

e.g if there are 10 input nodes there should be fewer nods in layer 1 and so on.

Question 7

Which parameters affect the learning

Answer 7

number of hidden layers, number of nodes in those layers, number of training iterations. Biasses

Question 8

What is a hyperparameter?

Answer 8

A hyperparameter is the parameters that affect learning, but now how it learns. So.. learning rate and regularization parameters.

Question 9

What are the performance measures for regression models?

Answer 9

RMSE or Mean Absolute error.

Question 10

What is the performance measures for classification

Answer 10

Accuracy
F1 scores
Youden’s Index
precision

Question 11

What is the accuracy performance measure for classification models?

Answer 11

It measures how accurate your model is at predicting and classifying correctly.

“The proportion of true results among the total number of cases”

Question 12

What is the issue with accuracy performance measure?

Answer 12

It might not be useful.

If we predict how many asteroids we can just say zero and be right 99 % of the time. But the model is not very useful then.

Question 13

which activation functions would you use for classification?

Answer 13

sigmoid or tanh

Question 14

When is accuracy a good performance measure to use?

Answer 14

When we want to classify almost evenly distributed categories.

It does not make sense to use in very imbalanced cases.

Question 15

What is the precision performance measure?

Answer 15

It is how often your model predicts truly correct.

How often your model predicts a case correctly

Question 16

When do we use the precision performance measure?

Answer 16

When we want to be very precise with our predictions.
when it is important that the cases that are predicted are right. compared to when you just want to be statistically right

Question 17

What is the F1 performance measure?

Answer 17

The F1 measure is the harmonic mean.

It considers the precision, sensitivity and accuracy

Question 18

What is the accuracy and precision of the following classification, in the case 99 % of asteroids will miss the earth

Our model predicts that no asteroids will hit the earth

Answer 18

The accuracy is 99 %.

The precision is 0 - because there are no cases correctly predicted. e.g. measured a specific asteroid and then predicting that case.

We have not successfully predicted the true positives.

Question 19

If you are a police inspector and you want to catch criminals, you want to be sure that the person you catch is a criminal (Precision) and you also want to capture as many criminals (Recall) as possible. The F1 score manages this tradeoff.

Answer 19

okay thanks

Question 20

What is Youden’s Index?

Answer 20

It is a performance measure that takes into account specificity and sensitivity. It can yield 1 or 0.

1 when there is no false positives or false negatives. (perfect test)
0 when the test gives equal weight to positive and negative results (equal amount of wrong and right predictions.)