04_supervised concepts

Question 1

What is the goal for supervised problems?

Answer 1

find function (task)
that relates input data (x)
to output data (y)
with hyperparameters (θ)

such that f(x;θ) = y

Question 2

What is a hyperparameter?

Answer 2

a model parameter that the model does not learn (given by programmer)

Question 3

What is the traditional (rule-based) approach to a supervised learning setup?

Answer 3

The person has seen x and seen y, writes the function

gives the function to the model which tests with unseen x and unseen y

Question 4

What is a machine-learning approach to a supervised learning setup?

Answer 4

the model finds the rules (function f) by itself

Question 5

What tasks can ML learn?

Answer 5

• (multi-class) Classification
• binary classification
• multi-label classification
• regression
• object detection
• semantic segmentation
• instance segmentation
• synthesis

Question 6

What is multi-class classification?

Answer 6

mapping input feature to discrete classes of a single label
(for one label multiple classes)

eg:
label: color
classes: red, green, blue

Question 7

What is binary classification?

Answer 7

mapping input features to a binary label

eg:
label: status
two classes: on, off

Question 8

What is multi-label classification?

Answer 8

mapping input features to discrete classes of multiple labels (with multiple classes)

eg:
labels: color, sort, quality
with multiple classes per label,
eg red/green/blue for color or good/medium/bad for quality

Question 9

What is regression?

Answer 9

mapping input features to continuous variable

eg x = time, y = value

Question 10

What is object detection?

Answer 10

approximately localize features in image data with bounding boxes

eg: boxes in picture for cat and dog

Question 11

What types of segmentation are there?

Answer 11

• semantic segmentation
• instance segmentation

Question 12

What types of classification are there?

Answer 12

• multi-class classification
• binary classification
• multi-label classification

Question 13

What is semantic segmentation?

Answer 13

assign class label to each pixel of an image based on what it is showing

eg what is part of cat? what is part of dog?

Question 14

What is instance segmentation?

Answer 14

assign class label to each pixel of an image based on what it is showing
AND discriminate different instances of the class

eg class animal, identifies two instances of class animal

Question 15

What is synthesis?

Answer 15

generate new data points based on a learned distribution

eg StyleGAN2 (creates faces) or Style Transfer

Question 16

What is iid?

Answer 16

independent and identically distributed data

–> individual samples in both data sets are produced by the same data generation process

Question 17

What is assumed when running an ML model on previously unseen data?

Answer 17

that unseen (new) data and the already seen (training) data are iid

–> the individual samples in both data sets are produced by the same data generation process

Question 18

What is the lesson in iid?

Answer 18

for small sample sizes, data sets that are iid may still differ significantly!

if sample is big, they appear more identical

Question 19

How will the distributions in real data sets look like?

What are the implications on the performance of the model?

Answer 19

will look different despite being iid, because of the limited extent (size)

successful training on one data set does not imply good performance on unseen data!
–> therefore, the model has to generalize well by preventing overfitting

Question 20

What is a decision boundary?

Answer 20

the decision boundary separates the different classes
as learned by the trained model

Question 21

What is overfitting?

Answer 21

the model memorizes the structure of the training/seen data
–> as a result, it generalizes badly on the overall data distribution

Question 22

What can we do against overfitting?

Answer 22

regularization methods

Question 23

When does a model generalize well?

Answer 23

when the decision boundary leads to equally good performance on both data sets

Question 24

How do we measure, if the model generalizes well?

Answer 24

We have to define a performance metric

Question 25

What does a performance metric do?

Answer 25

provide a quantitative assessment of how well our model performs

Question 26

Base on what does the performance metric provide an assessment on the model performance?

Answer 26

• ML implementation (model type and loss function)
• dataset
• task to be learned

Question 27

What does the Performance metric need to be?

Answer 27

tailored to the model
and the task
and the dataset

Question 28

How can we identify overfitting?

Answer 28

By comparing the performance on the seen/training data and some previously unseen/test data

Question 29

What is usually done to the existing data in supervised learning?

Answer 29

the data is randomly split into three parts:

1) Training data set
2) Validation data set (hyperparameter-tuning)
3) Test data set (evaluate model performance)

Question 30

What are typical ratios for splitting the dataset?

Answer 30

Train: 0.7
Validate: 0.15
Test: 0.15

Question 31

What is a stratified split?

Answer 31

If we have more than one class, all the classes are split into train validate test on their own and according to the set ratios

–> preserves the class fractions in split data sets

Question 32

How can the data be validated if the data set available is not big enough to perform a meaningful split?

Answer 32

“recycle” data by using cross-validation to get a better estimate

Question 33

How is the k-fold cross-validation carried out?

Answer 33

1) split shuffled data set into three parts
2) train with two parts and test with the third
3) repeat with reassigning test and train
–> in independent runs
4) results in k performance metrics
–> report: avg + std / best-of k

Question 34

Does the k-fold cross-validation improve the models performance?

Answer 34

No

Because it uses independent runs, which means that they don’t build up on each other

it just gives a more reliable estimate of the model’s performance

Question 35

What are methods to force the model to generalize? (6)

Answer 35

• limiting model capacity
• introducing uncertainty
• dropout (only NN)
• introducing noise
• early stopping (only NN)
• Bagging / Ensembling

Question 36

What is the usual general supervised learning pipeline?

Answer 36

1) feature engineering
2) data scaling
3) data splitting
4) define hyperparameters
5) train model on training data for fixed hyperparameters
6) evaluate model on validation data
7) repeat 4 to 6 until performance on validation data maximised
8) evaluate trained model on test data and report the test data performance

–> performance metrics should be similar between test and validation data before showing the model the test data

Question 37

How do we measure the performance of our model?

Answer 37

Benchmarking

Question 38

What is benchmarking?

Answer 38

refers to the process of quantitatively assessing your ML model’s performance

Question 39

What is a metric?

Answer 39

measure for performance, depends on the task and the data set

Question 40

What are the two most important regression task metrics?

What is the intuition behind the calculations?

Answer 40

• MAE (mean absolute error)
  1/N Summe I yi’ -y^iI
• RMSE (root mean square error)
  Wurzel 1/N Summe (yi’ - y^i)^2

Intuition: by how much deviates your model prediction from the ground-truth on average

Question 41

What is the difference between MAE and RMSE for small datasets?

Answer 41

RMSE is more sensitive to outliers

it depends on the model and the problem if this is beneficial or not

Question 42

What are (binary) classification metrics?

Answer 42

• Accuracy
• Precision
• Recall

Question 43

What is the Accuracy Metric and how is it calculated?

Answer 43

What is the overall fraction of correct predictions?

Accuracy = (TP + TN) / (TN + TP + FP + FN)

we correctly identified 95% of all dogs in the image

Question 44

What is the Precision Metric and how is it calculated?

Answer 44

What fraction of our positive predictions is truly positive? (quantifies “correctness”)

Precision = TP / (TP + FP)

95% of the dogs we predicted are actual dogs

Question 45

What is the Recall Metric and how is it calculated?

Answer 45

What fraction of actual positives has been identified? (quantifies “completeness”)

Recall = TP / (TP + FN)

we correctly found 95% of the dogs that are in the image

Question 46

What is the issue with Accuracy/Precision/Recall as Classification Metrics?

Answer 46

Class imbalance!

eg “Will this asteroid impact Earth”
here, Recall is the most important because we don’t want to miss any hitting asteroids

Question 47

What is the confusion matrix?

Answer 47

a common way to visualize the performance of a classification model

provides information on systematic confusion learned by the classifier

all elements in one row must sum up to unity

Question 48

What is a sign for a well-trained classifier regarding the confusion matrix?

Answer 48

the diagonal values should be as high as possible while off-diagonal elements should be as low as possible

Question 49

Is accuracy a good metric for object detection?

Answer 49

No

would be very high by looks at all individual pixels. no indication if the correct pixels were identified correctly

Question 50

What is a good metric for object detection and image segmentation?

Answer 50

Intersection over union metric

Question 51

What is IoU and how is it calculated?

Answer 51

Intersection over union metric

IoU = intersection / union = (AnB)/(AuB)

A is prediction-Shape, B is ground-truth

Question 52

Where is the IoU undefined?

Answer 52

Where there is no ground truth

Question 53

How to report metrics?

Answer 53

• individual metric
• best-of-n (eg with cross-validation)
• averaging results (average metric over n model runs + standard deviation)

best choice depends on the specific problem and use case