ML-04 - Neural network

Question 1

ML-04-Neural network

Why use ANN over polynomial regression?

Answer 1

For a large number of features, polynomial regression gets too big.

Ex:
1,000 raw features with quadratic features x_I^2 will have ~1,000,000 input features. Scales even worse for cubic+.

Question 2

ML-04-Neural network

Where is the dendrite(s) located? (See image)

Answer 2

(See image)

Question 3

ML-04-Neural network

Where is the nucleus located? (See image)

Answer 3

(See image)

Question 4

ML-04-Neural network

Where is the axon located? (See image)

Answer 4

(See image)

Question 5

ML-04-Neural network

Where are the input wires located? (See image)

Answer 5

(See image)

Question 6

ML-04-Neural network

Where is the cell body located? (See image)

Answer 6

(See image)

Question 7

ML-04-Neural network

Where is the output wire located? (See image)

Answer 7

(See image)

Question 8

ML-04-Neural network

Where is the node of ranvier located? (See image)

Answer 8

(See image)

Question 9

ML-04-Neural network

Where is the axon terminal located? (See image)

Answer 9

(See image)

Question 10

ML-04-Neural network

Where is the myelin sheath located? (See image)

Answer 10

(See image)

Question 11

ML-04-Neural network

Where is the Scwann cell located? (See image)

Answer 11

(See image)

Question 12

ML-04-Neural network

What is the difference between a NN and the perceptron?

Answer 12

• Perceptron uses step function
• Perceptron outputs are binary, i.e. in {0, 1}.
• NN can use other activation functions.
• NN outputs are real values, often in [0, 1] or [-1, 1]

Question 13

ML-04-Neural network

What notation would you use to denote which layer a weight belongs to?

Answer 13

w_3^(1) = first layer, 3rd neuron
This connects input values (layer 1) with the 2nd layer.
(See image)

Question 14

ML-04-Neural network

What notation would you use to denote the 3rd neuron’s activation in the 2nd layer?

Answer 14

a_3^(2) = first layer, 3rd neuron
(See image)

Question 15

ML-04-Neural network

Describe what the numbers mean in this picture (See image)

Answer 15

• Red is the layer
• Blue is which neuron in the layer it is or which neuron the weight belongs to.
• Green is which specific weight it is.

Question 16

ML-04-Neural network

In a neural network, what is “a_1^(2)”?

Answer 16

The activation for the 1st neuron in the 2nd layer, i.e.

a = g(z) = g(W^T * x)

Question 17

ML-04-Neural network

In a neural network, what is z_1^(2)?

Answer 17

The weighted summation of the inputs for the 1st neuron in the 2nd layer.

z = W^T * x

Question 18

ML-04-Neural network

What is the difference between cross-entropy and sparse cross-entropy?

Answer 18

• CE uses one-hot coded data.
• SCE uses integer labels.

Question 19

ML-04-Neural network

Why use sparse cross-entropy loss?

Answer 19

Saves memory and computation when labels are sparse.

Question 20

ML-04-Neural network

How can you use a neural network for linear regression?`

Answer 20

Single layer network with activation function g(x) = x.

Question 21

ML-04-Neural network

What’s the requirement for an activation function?

Answer 21

Must be differentiable.

Question 22

ML-04-Neural network

Should you calculate regularization for the bias term?

Answer 22

No.

Question 23

ML-04-Neural network

How do you check is backpropagation is correctly implemeted?

Answer 23

Gradient checking.

Question 24

ML-04-Neural network

What is gradient checking?

Answer 24

Numerically checking if the gradient matches the analytical version. Roughly similar values means it should be okay.

Question 25

ML-04-Neural network

What’s the formula for gradient checking?

Answer 25

(See image)

Question 26

ML-04-Neural network

What’s a problem you might face when initializing weights too large?

Answer 26

Exploding gradients problem.

Question 27

ML-04-Neural network

What’s a problem you might face when initializing weights too small?

Answer 27

Vanishing gradients problem.

Question 28

ML-04-Neural network

What causes vanishing/exploding gradients?

Answer 28

The recursive nature of backpropagation.

Question 29

ML-04-Neural network

How does Xavier initialization work?

Answer 29

• Weights: Layer l is normally sampled from mean 0 and variance 1/(n^(l-1)).
• Bias: Zero

Question 30

ML-04-Neural network

What output activation function would you use for binary classification?

Answer 30

Sigmoid

Question 31

ML-04-Neural network

What output activation function would you use for multiclass classification?

Answer 31

Softmax

Question 32

ML-04-Neural network

What output activation function would you use for multi-label classification?

Answer 32

Sigmoid

Question 33

ML-04-Neural network

What output activation function would you use for regression?

Answer 33

A linear activation function.

Question 34

ML-04-Neural network

What loss function would you use for binary classification?

Answer 34

Binary cross entropy

Question 35

ML-04-Neural network

What loss function would you use for multiclass classification?

Answer 35

Categorical cross-entropy

Question 36

ML-04-Neural network

What loss function would you use for multi-label classification?

Answer 36

Binary cross-entropy

Question 37

ML-04-Neural network

What loss function would you use for regression?

Answer 37

MSE, MAE ++