Ch4 Training a Digit Classifier End-of-Chapter Questions

Question 1

What is a mini-batch?

Answer 1

A small group of inputs and labels gathered together in two arrays. A gradient descent step is updated on this batch (rather than a whole epoch)`

Question 2

What is a “forward pass”?

Answer 2

Applying the model to some input and computing the predictions.

Question 3

How is a greyscale image represented on a computer? How about a color image?

Answer 3

Greyscale images are represented by a matrix of pixel intensities with 0 representing white and 255 representing black. Shades of grey are numbers in between. A color image is represented by a set of 3 matrices, each representing a pixel’s intensity of red, green, and blue.

Question 4

What is a rank-3 tensor?

Answer 4

It’s a tensor with 3 axes or dimensions

Question 5

What is the difference between tensor rank and shape? How do you get the rank from the shape?

Answer 5

Tensor rank refers to the number of axes in a tensor. Tensor shape refers to the length of each axis. You can get the rank from the shape by using the length function: len(tensorname.shape)

Question 6

What are RMSE and L1 norm?

Answer 6

RMSE - root mean square error. Take the mean of the squared difference and then take the square root of the answer. It penalizes bigger mistakes more heavily than L1 norm.

L1 norm - mean absolute difference. Take the mean of the absolute value of differences

Question 7

What is broadcasting?

Answer 7

When trying to do a mathematical operation between tensors of different ranks, broadcasting expands the tensor with the smaller rank to have the same size as the one with the larger rank.

Question 8

What is SGD?

Answer 8

Stochastic Gradient Descent. It is an iterative algorithm that starts from a random point on a function and travels down its slope in steps until it reaches the lowest point of that function.

Question 9

Why does SGD use mini-batches?

Answer 9

Instead of calculating the loss over all data items, which would take forever, SGD calculates the loss over a portion of data items at a time. This increases computational speed because it reduces the number of required calculations (derivatives).

Question 10

What are the 7 steps for SGD in machine learning?

Answer 10

7 Steps for SGD:

1. Initialize the weights
2. Make predictions using model with these weights
3. Calculate the loss based on these predictions
4. Calculate the gradient, which measures for each weight how changing that weight would change the loss.
5. Step (change) all the weights based on that calculation
6. Go back to step 2 and repeat the process
7. Stop the training process

Question 11

How do we initialize the weights in a model?

Answer 11

Use random numbers.

Question 12

What is loss?

Answer 12

Loss is value that represents how well (or badly) our model is doing.

Question 13

What can’t we always use a high learning rate?

Answer 13

A learning rate that is too high results in large steps that may miss the minimum loss, leading to the loss getting worse or bouncing around rather than converging at the minimum.

Question 14

What is a gradient?

Answer 14

A gradient is a derivative of the loss with respect to a parameter of the model.

Question 15

Why can’t we use accuracy as a loss function?

Answer 15

The gradient of a function is its slope–how much the value of the function changes divided by how much we changed the input:

(y_new - y_old)/(x_new - x_old)

The problem is that a small change in weight (x) isn’t likely to cause the prediction to change, so (y_new - y_old) will almost always be 0, i.e., the gradient is 0 almost everywhere. Thus, a small change in weight will often not change the accuracy at all. If the gradient is 0, the model can’t learn from that step. We need a function that can show differences from small changes in weights.

Question 16

What is special about the shape of the sigmoid function?

Answer 16

It looks like an “S”. It can take any input value, positive or negative, and output a value between 0 and 1.

Question 17

What is the function to calculate new weights using a learning rate?

Answer 17

w -= w.grad * lr

Question 18

What does the DataLoader class do?

Answer 18

It takes in a dataset, shuffles it on every epoch and creates mini-batches. It returns an iterator over the batches.

Question 19

Write pseudocode showing the basic steps taken in each epoch for SGD.

Answer 19

for features, targets in a minibatch:

• preds = model(features)
• loss = loss_function(preds, targets)
• calculate the gradients for each parameter
• update the parameters by subtracting gradient * learning_rate
• reset gradients to zero for each parameter

Each epoch goes through all the minibatches.

Question 20

Create a function that if passed two arguments [1,2,3,4] and ‘abcd’, returns [(1, ‘a’), (2,’b’), (3,’c’), (4,’d’)]. What is special about that output data structure (as related to PyTorch Datasets)?

Answer 20

list(zip([1,2,3,4], ‘abcd’))

This is the format of a Dataset in PyTorch: a collection that contains tuples of independent and dependent variables.

Question 21

What does view do in Pytorch?

Answer 21

It changes the shape of a tensor without changing its contents

Question 22

What are the bias parameters in a neural network? Why do we need them?

Answer 22

If we only used the function x*weights, the function would always be equal to zero when x = 0. No matter how we change the weight, the function would always be equal to zero. We need something a little more flexible, so we add bias to the equation.

Question 23

What does the @ opertator do in Python

Answer 23

Matrix multiplication

Question 24

What does the backward method do?

Answer 24

It calculates the gradients.

Question 25

Why do we have to zero the gradients?

Answer 25

backward will add the calculated gradients to any gradients that are already stored. If you want to calculate gradients starting from zero, you have to zero out any gradients stored in the parameters.

Question 26

What information do we have to pass to Learner (5 key things)?

Answer 26

DataLoaders (train and test data), model, optimization function, loss function, metrics

Question 27

What is ReLU?

Answer 27

Rectified Linear Unit. It’s an activation function that transforms any negative numbers into zero.

Question 28

What is an activation function?

Answer 28

It’s a nonlinear function

Question 29

What’s the difference betwen F.relu and nn.ReLu?

Answer 29

They represent the same thing but F.relu is a function and nn.ReLu is a PyTorch module.

Question 30

The universal approximation theorem shows that any function can be approximated as closely as needed using just one nonlinearity. So why do we normally use more?

Answer 30

With more layers, we train the model more quickly and it will take up less memory because we don’t need to use as many parameters with deeper models.

Question 31

What is a “backward pass”?

Answer 31

Computing the gradients of the loss with respect to all model parameters

Question 32

What is a gradient descent step?

Answer 32

Taking a step in the direction opposite to the gradients to make the model parameters a little bit better.

Question 33

What is the learning rate?

Answer 33

The size of the step we take when applying SGD to update the parameters of the model.