Chapter 4: Fundamentals of ML

Question 1

Four broad categories of ML

Answer 1

1. Supervised Learning
2. Unsupervised Learning
3. Self-Supervised Learning
4. Reinforcement Learning

Question 2

Why is validation data helpful?

Answer 2

helps you tune the hyperparamaters of the model. The number of layers, this size of layers etc.

Question 3

information leak

Answer 3

every time you tune your model using the validation data some information about the validation data leaks into your model

Question 4

Why don’t you evaluate ML models with the training data?

Answer 4

After a certain number of epochs (which you can’t predict ahead of time) the model will start to overfit to the training data

Question 5

How do you evaluate a ML model?

Answer 5

By splitting off some data to evaluate it on called validation data

Question 6

What is the goal when training machine learning models? models that do what?

Answer 6

Generalize well

Question 7

When you evaluate machine learning models, what are you evaluating?

Answer 7

Their ability to generalize

Question 8

How many sets of data should you use to train and evaluate a model?

Answer 8

3. Train, Validate, Test

Question 9

hyperparameters vs parameters

Answer 9

hyperparameters = number or size of layers in neural network
parameters = the weights of each layer

Question 10

Why does developing a neural network require the number of data sets it does?

Answer 10

Because training data sets the parameters of the model, and validation data tunes the hyperparameters

Question 11

How are the hyperparameters of the model tuned?

Answer 11

using the performance of the model on the validation dat as a feedback signal

Question 12

Classic model evaluation recipes

Answer 12

simple hold-out validation, K-fold val, iterated K-fold val with shuffling

Question 13

Simple hold-out validation

Answer 13

set out test, validation and training

Question 14

What might stop you from using simple hold-out validation?

Answer 14

If there’s too little data available. You can check this if different rounds of shuffling before splitting produce very different model performance

Question 15

K-fold cross Validation

Answer 15

Split data into some K number of equal sets, then use all but 1 set to train the data and evaluate the data on the held-out partition. You take the average of the evaluation score as the model evaluation score. Models discarded after generating evaluation score

Question 16

Iterated K-fold cross Validation with shuffling

Answer 16

Applying K-fold cross validation multiple times and shuffling the the data before each new split. The final score is the average of the scores from each run of the K-fold validation

Question 17

How do you evaluate your model if you have very little data?

Answer 17

Iterated K-fold cross Validation with shuffling

Question 18

How should you evaluate your model if it is exhibiting considerable variance in train test split?

Answer 18

K-fold cross validation

Question 19

How can you help ensure Data Representativeness in your evaluation?

Answer 19

Using data shuffling before you split

Question 20

Why can redundancy in your data be a problem for validation?

Answer 20

Because if there are repeats and then get split into test and training data, your model is then partially trained on your test data

Question 21

What are some examples of data preprocessing?

Answer 21

Vectorization, Normalization, handling missing values, feature extraction

Question 22

data vectorization

Answer 22

all inputs and targets in a neural network must be tensors of floating-point data

Question 23

value normalization

Answer 23

have to make sure each feature is normalized so that its mean is 0 and its standard deviation is 1 is a common method but isn’t always strictly necessary

Question 24

feature engineering

Answer 24

process by which a human makes a problem easier for a neural network using their deep understanding to explress the problem in a simpler way

Question 25

Why can good feature engineering still be helpful for deep learning models?

Answer 25

lets you solve the problem with far less data

Question 26

What is the fundamental tension in machine learning?

Answer 26

The tension between optimization and generalization

Question 27

Underfit model

Answer 27

When there is still progress to be made in generalization of the model via optimization on the training data

Question 28

Best way to prevent a model from learning irrelevant or misleading patterns in the training data?

Answer 28

Get more training data

Question 29

How can you fight overfitting with limited data?

Answer 29

put constraints on what information the model is able to store or how much it is able to store

Question 30

Regularization

Answer 30

The process of fighting overfitting by putting constraints on the information a model learns

Question 31

A model’s capacity

Answer 31

The number of learnable parameters in a model

Question 32

What is the simplest way to prevent overfitting via regularization?

Answer 32

by reducing the size of the model: the number of learnable parameters in the model

Question 33

Why does reducing the memorization capacity of a network help prevent overfitting?

Answer 33

It won’t be able to learn mapping as easily so to minimize loss it will have to resort to compressed representation that maximize the predictive power

Question 34

How does model capacity affect its losses?

Answer 34

Bigger networks will minimize their training loss much faster as they fit to the training data but their validation loss will also be much bigger as they start overfitting much earlier

Question 35

Weight regularization

Answer 35

adding a cost to the loss function of a network associated with having large weights

Question 36

L1 regularization

Answer 36

the cost added to the loss function by large weight values is proportional to the absolute value of the weight coefficients (the L1 norm of the weights)

Question 37

L2 regularization

Answer 37

the cost added is proportional to the square of the value of the weight coefficients (the L2 norm of the weights). Also called weight decay

Question 38

Dropout

Answer 38

One of the most common and effective regularization techniques. randomly dropping some a number of output features in a layer

Question 39

Success metric for balanced-classification problems?

Answer 39

Accuracy and area under the receiving operating characterister curve (ROC AUC)

Question 40

Success metric for imbalanced-classification problems?

Answer 40

Precision, Recall

Question 41

Success metric for ranking problems and multilabel classification?

Answer 41

Mean average precision

Question 42

How should data be formatted for a neural network

Answer 42

1. Data in tensors
2. Data should be scaled to small values
3. If Data is heterogeneous should be normalized
4. Often want to do some feature engineering

Question 43

Statistical Power

Answer 43

A model that is capable of beating a dumb baseline

Question 44

requirements for a loss function?

Answer 44

1. needs to be computable given only a mini-batch of data

2. function needs to be differentiable (so you can use backpropagation to train your model)

Question 45

common last-layer activations

Answer 45

Softmax, Sigmoid, none (for regression)

Question 46

common loss functions

Answer 46

Binary crossentropy, categorical crossentropy, MSE

Question 47

How do you know when you’ve reaced overfitting?

Answer 47

When the model’s performance on the validation score starts to degrade

Question 48

Steps in building neural network

Answer 48

1. Define problem; assemble dataset
2. Choose measure of success
3. Deciding on an evaluation protocol
4. preparing your data
5. Last-layer act, loss function, optimization config
6. Scale up to overfitting
7. Regularize model; Tune hyperparameters

Question 49

final step before testing on test data?

Answer 49

Evaluate your validation procedure by combing training and validation data. If score is significantly worse than on the validation data, your validation process wasn’t reliable or your began to overfit to your validation data