Classification and Machine Learning Evaluation

Question 1

What is classification?

Answer 1

This is when you find patterns in input data and divide it into categories

Question 2

What is regression?

Answer 2

Building a model to solve a problem

Question 3

Define clustering?

Answer 3

This is where data which is in different categories forms clusters (groups) when graphed

Question 4

What is the machine learning process? (5 steps)

Answer 4

1. Data collection
2. Feature selection
3. Algorithm Choice
4. Training
5. Evaluation

Question 5

What is overfitting?

Answer 5

A model overfits when it describes the randomness associated with the data, rather than the underlying relationship between the data points

Question 6

What is underfitting?

Answer 6

This is when the model is too simple to understand the complex problem (so it is no so accurate)

Question 7

What is the rule with under/over fitting (occam’s razor)

Answer 7

We should use the simplest model unless we have to use a more complex one

Question 8

What are the 3 types of data set and what are they each used for?

Answer 8

> Training data: Used to train the model

> Validation data: Used to evaluate the different models that we have created

> Test data: Use to test the bench mark the model at the end

[Picture 1]

Question 9

What is important about the data for the 3 types of data?

Answer 9

The training, validation and test data must not overlap. (contain any of the same data)

Question 10

What can happen with regards to the training data if we make the model too complex?

Answer 10

It doesnt learn the underlying principle, it just starts to learn the data set.

Question 11

Is it possible to get the error during training to 0? Would we want to do this? why?

Answer 11

Yes it is possible. No we do not want to do this because the AI would just be learning the data set

Question 12

When do we stop training a machine learning AI? why?

Answer 12

We want to stop when the error of the validation data set starts to increase. The error will increase because the model is starting to over fit

[Picture 2]

Question 13

What happens if you dont stop training a machine learning AI?

Answer 13

The model would become too complex and it would start to overfit

Question 14

What is cross validation?

Answer 14

> When there is not enough data to create three sets large enough, cross validation is a common way to test the learned model on more data points.

> The data is split into several batches

> Each batch is tested in turn whilst others are used for training

Question 15

What is a binary classifier?

Answer 15

This is a method used to evaluate a class. You form the following diagram:

[Picture 3]

Question 16

How is accuracy predicted with a binary classifier? [Picture 3]

Answer 16

Accuracy % = 100 × (#TP+#FP) / (#TP + #TN + #FP + #FN)

Question 17

How is left collumn sensitivity calculated? [Picture 3]

Answer 17

Sensitivity % = 100 × #TP / (#TP + #FN)

Question 18

How is right collumn sensitivity calculated? [Picture 3]

Answer 18

Sensitivity % = 100 × #TN / (#TN + #FP)

Question 19

What is the equation for recall? [Picture 3]

Answer 19

Recall % = 100 × #TP / (#TP + #FN)

Question 20

What is recall also?

Answer 20

Recall = Left collumn Sensitivity

Question 21

How is top row precision calculated? [Picture 3]

Answer 21

Precision % = 100 × #TP / (#TP + #FP)

Question 22

How is bottom row precision calculated? [Picture 3]

Answer 22

Precision % = 100 × #TN / (#TN + #FN)

Question 23

Which is the least informative method of evaluation?

Answer 23

Accuracy alone

Question 24

What is the equation for F1?

Answer 24

F1 = 2 × (precision × recall) / (precision + recall)

Question 25

What is MCC (not the equation)?

Answer 25

Matthew’s Correlations Coefficient

Question 26

What is the equation for MCC?

Answer 26

MCC = (#TP × #TN - #FP × #TN) / √ ((#TP + #FP) (#TP + #FN) (#TN + #FP) (#TN + #FN))

Question 27

What is the benefit of MCC?

Answer 27

It is a better measurement when the data set is unbalanced

Question 28

What is the purpose of the confusion matrix?

Answer 28

It is a convenient way to represent the accuracy of a multi-class (non-binary) classifiers. It is like a heat map

[Picture 4]

Question 29

What are the axis of the confusion matrix?

Answer 29

Each entry at coordinate (x,y) in the matrix corresponds to the number of elements of class x classified as y

[Picture 4]

Question 30

What is the ideal result of a confusion matrix (best classification)?

Answer 30

All the elements should be diagonal [Picture 4]

Question 31

What is a ROC curve?

Answer 31

A convenient way to compare different models in the receiver operator characteristic

Question 32

What does a diagonal classifier of a ROC curve mean?

Answer 32

Along the line, there is a 50% chance of being correct and 50% chance of being incorrect

Question 33

What is the perfect classifier on an ROC curve?

Answer 33

The perfect classifier would be in the top left corner (0,100)

Question 34

On an ROC curve, which line is best?

Answer 34

The one with the largest area under it

Question 35

What is a parametric method?

Answer 35

This is when you decide what parameters and how many the AI is going to learn before training starts

Question 36

What happens to the training data after the AI has been training with a parametric method?

Answer 36

After the rules have been learned by the AI the data can be discarded

Question 37

What are some issues with the parametric method?

Answer 37

We cannot always separate everything with a single hyperplane

Question 38

What is a non-parametric method?

Answer 38

There are no defined parameters at the start. The parameters will be learned as training happens. This method focusses on the data rather than a particular structure