Chapter3: Machine Learning Experiments

Question 1

Describe the accuracy of a classification model

Answer 1

correct / real

Question 2

describe the error of a classification model

Answer 2

incorrect / real

Question 3

what is the problem with error and accuracy? what is a better alternative?

Answer 3

it is unreliable for imbalanced data

confusion matrix

Question 4

how can we compute accuracy from a confusion matrix

Answer 4

total

Question 5

what does precision and recall measure about a classifier

Answer 5

its ability to classify positive samples

Question 6

recall =

Answer 6

TP + FN

Question 7

Precision =

Answer 7

TP + FP

Question 8

what is the f_1 score

Answer 8

precision + recall

Question 9

what is specificity

Answer 9

measure for assessing a classifiers ability to classify negative samples

Question 10

specificity =

Answer 10

TN + FP

Question 11

1 - specificity is otherwise known as

Answer 11

false positive rate

Question 12

what is ROC analysis

Answer 12

applies to binary classifiers. we plot sensitivity(true positive rate) and 1 - specificity.

we want the area under the curve to be as close to 1 as possible

Question 13

what is error in a regression model

Answer 13

difference between predicted and desired output

Question 14

list types of error for a regression model

Answer 14

root mean square error
mean absolute error
mean absolute percentage error
sum of squares error

Question 15

what is the coefficient of determination

Answer 15

R^2 score in single output case.

sum of squares error / sum to n (y - 1/n sum to n y) ^2

Question 16

what is sample error

Answer 16

the error computed using a performance metric from a set of samples

Question 17

what is true error

Answer 17

the probability a random sample is misclassified

Question 18

how is true and sample error different in regression

Answer 18

it is the expectation of the error

Question 19

how do we get bias and variance values

Answer 19

from the expected squared prediction error

Question 20

what is bias error

Answer 20

(y - E(f))^2

repeat with different sets of training data and measure how true they are

Question 21

what is variance error

Answer 21

E[(f - E[f])^2]

repeat with different sets of training data and measure how much prediction varies

Question 22

what is overfitting

Answer 22

model is over complex. Low bias high variance

Question 23

what is underfitting

Answer 23

model is too simple. high bias, low variance

Question 24

what is a confidence interval

Answer 24

how good of an estimate of true error is provided by sample error

Question 25

what is a z test

Answer 25

compares two classifiers

Question 26

what is the confidence interval of a classifier

Answer 26

a range [error -a, error + a ]

a = zp root(error * (1 - error) / n)

Question 27

describe the steps of a z test

Answer 27

1. calculate zp = d / o
   where d = error A - error B
   o = root (zpa + zpb)
2. get p using table
3. c = 1 - (1-p /2)

Question 28

what is a hypothesis

Answer 28

a model trained on a sample set

Question 29

how do we evaluate a model

Answer 29

multiple train-test trials and average these error rates

Question 30

what methods for data splitting are there

Answer 30

holdout
random subsampling
k fold cross validation
leave one out
bootstrap

Question 31

what is holdout

Answer 31

we have a data set split into a single test and train division

Question 32

what is random subsampling

Answer 32

have k splits of a number of test samples, chosen randomly. the rest are training samples

Question 33

what is k fold cross validation

Answer 33

we split the data into k partitions. all examples are used in train and test, but only in a test set once.

low k = not enough trails, high k = small test set -> high variance

Question 34

what is Leave one out

Answer 34

like k fold cross validation but k = n, so only one sample in the test set each time

Question 35

what is bootstrap

Answer 35

randomly select m samples, use these for training. use the rest for testing

Question 36

what is hyperparameter selection

Answer 36

choose the model with the least error.