model evaluation

Question 1

What is the requirement for a goodness metric for backpropagation to work?

Answer 1

has to be differentiable

Question 2

How do you define the goodness of your model?

Answer 2

Ability to generalize to unseen data

Question 3

What can affect generalizability?

Answer 3

• algorithm
• hyperparameter values
• training data
• random initialization (can affect accuracy)

Question 4

What are 4 methods of quantifying generalization goodness?

Answer 4

1) accuracy, precision, recall
2) mean absolute error
3) RMSE
4) area under ROC curve

Question 5

Come up with a metric for Predicting lung cancer from chest x-rays

Answer 5

Recall, precision, FN/P

Question 6

Come up with a metric for predicting high school GPA

Answer 6

MAE

Question 7

Come up with a metric for evaluating search engine results

Answer 7

recall

Question 8

Come up with a metric for predicting the location of an object in 3D space

Answer 8

euclidean/cosine distance

Question 9

Come up with a metric for predicting if twitter user is a liberal or conservative

Answer 9

AUC

Question 10

What is sensitivity?

Answer 10

True positive rate, also recall

Question 11

What is specificity?

Answer 11

True negative rate

Question 12

What is precision?

Answer 12

A measure of exactness. The percentage of tuples labeled as positive and are actually such

Question 13

What is the F-measure?

Answer 13

Harmonic mean of precision and recall, giving equal weight to each

Question 14

What metrics can you use for regression?

Answer 14

MAE, RMSE

Question 15

Why do we take the square of metrics?

Answer 15

Taking the square results in steeper gradients when searching and gives higher penalities. RMSE accentuates the impact of outliers. Otherwise might take too long t

Question 16

What is a condition of the AUC

Answer 16

Binary prediction metric that only applies when label is binary, predictions have to be probablistic.

Question 17

What is the AUC?

Answer 17

Probability that a randomly chosen positive label prediction will be greater than a randomly chosen negative label prediction

Question 18

What does an AUC of 0.5 mean?

Answer 18

AUC = 0.50 means no better than random chance

Question 19

What is K-fold CV?

Answer 19

Tuples randomly partitioned into K mutually exclusive subsets

Question 20

What is a standard value for K?

Answer 20

5 or 10

Question 21

What is the algorithm for K-fold CV?

Answer 21

1) Choose a K and randomly assign each row to a single fold (between 1 and K)
- This fold assignment indicates the fold in which that row will serve as a tuple in the test set
2) Conduct K phases of training/testing, start with fold 1, which will serve as the test set, with the other folds serving as the training set
3) Calculate error metric on the whole dataset (concatenation of results from each fold)

Question 22

Do you calculate an error metric for each fold in K-fold CV?

Answer 22

No, it is still only calculated once

Question 23

What is CV used for?

Answer 23

Model selection

Question 24

When would CV NOT give you better results than handpicking your train and test data?

Answer 24

With temporal data

Question 25

What is ROC curve?

Answer 25

Shows tradeoff between TPR and FPR. It allows us to visualize the tradeoff between the rate at which it mistakenly identifies negative cases as positive for different portions of the test set