Deep Learning

Question 1

What is an activation function? Why is it important?

Answer 1

The activation function is the function the neuron applies to the weighted sum of its inputs. It is important because it is how you introduce nonlinearity, since the function can be anything.

Question 2

What is a step function?

Answer 2

A function that produces binary output based on some threshold.

Question 3

What is a perceptron?

Answer 3

A perceptron is a neuron with the step function z > 0.

Question 4

What does MLP stand for?

Answer 4

Multi-layer perceptron.

Question 5

What is a multi-layer perceptron?

Answer 5

It’s when you chain together multiple layers of logic gates built from perceptrons, to represent complex logical functions.

Question 6

What is “universality”?

Answer 6

If a network has universality, then it can approximate any continuous function arbitrarily with just one hidden layer, given enough units.

Question 7

The definition of universality talks about one hidden layer. If that’s the case, why would you want multiple layers?

Answer 7

You can represent things more compactly if you use multiple layers.

Question 8

What does “differentiable” mean?

Answer 8

A function is differentiable if you can find the slope of its tangent at any point.

Question 9

What is ReLU?

Answer 9

ReLU is an activation function that is 0 for x < 0 and x for x > 0.

Question 10

Why do we like to use ReLU instead of step function?

Answer 10

The step function has no useful gradient since it is just flat everywhere. ReLU does have a useful gradient.

Question 11

What is risk? What is empirical risk?

Answer 11

Risk is the actual expected loss over the entire real distribution of data. We don’t know the real distribution of data, we just have our training set, so we only have empirical risk (the expected loss over the training set distribution).

Question 12

What’s the point of the VC Dimension?

Answer 12

The VC dimension proves that the generalization bound is finite, even for an infinite hypothesis set. Generalization bound is risk <= empiricalRisk + error. And VC dimension says that error is finitely bounded.

Question 13

What is the objective function?

Answer 13

It’s the loss function that we want to minimize.

Question 14

What is the vanishing gradient problem?

Answer 14

It’s when the gradient goes to zero – which often happens because you’re multiplying a lot of tiny weights together.

Question 15

What is it called when the gradient goes to zero?

Answer 15

The vanishing gradient problem.

Question 16

What is “over-saturation”?

Answer 16

It’s when the activation function is in a flat region – such as < 0 for the ReLU – so the gradient is zero, which causes the vanishing gradient problem.

Question 17

Give three solutions to the vanishing gradient problem

Answer 17

Any of these four: Better initialization, Better activation function, Regularization to rescale the gradient, Gradient clipping

Question 18

What does it mean if a model is “non-identifiable”?

Answer 18

It means that the model can get the same minimum loss with multiple different settings of the weights.

Question 19

Why is it a problem is a model is non-identifiable?

Answer 19

If we were using weights to draw conclusions about the model, having two different sets of weights both claim to be optimal would make that difficult.

Question 20

What does it mean if a function is “convex”?

Answer 20

It means that it has a single minimum, and a bowl shape going towards that minimum. Specifically, it means that if you pick any two points and draw a line between them, that line won’t intersect the function.

Question 21

Are deep neural networks convex?

Answer 21

Absolutely not.

Question 22

Give three examples of problems caused by nonconvexity

Answer 22

Any of: Local minima, Saddle points, Cliff structures, Asymptotes

Question 23

Why don’t you want to just initialize all the weights to zero?

Answer 23

For ReLU, this yields zero gradients. In general, this means every hidden unit with the same activation function would have the same output.

Question 24

What is the most common way to initialize?

Answer 24

Sampling from a distribution

Question 25

What is “warm start”?

Answer 25

It’s where you train the model on a different objective just to get some initialization weights, and then switch to the actual objective.

Question 26

What is dropout regularization?

Answer 26

Randomly remove hidden units by hardcoding their activation to 0, randomly choosing in each iteration of gradient descent. This simulates a bunch of different models.

Question 27

What is ensemble regularization?

Answer 27

Train a bunch of different models and average their predictions.

Question 28

What is variational noise and why would you do it?

Answer 28

It’s when you add noise from a Gaussian distribution during the forward pass part of training. It’s a way to regularize.

Question 29

What is a convolution?

Answer 29

It’s a window-based operation where you combine two vectors to extract spatial or contextual information.

Question 30

What is the vector you use to do the convolution called?

Answer 30

The kernel

Question 31

What is the kernel, in the context of CNNs?

Answer 31

It’s the vector you use to define and apply the convolution to your input

Question 32

What does CNN stand for?

Answer 32

Convolutional Neural Network

Question 33

What is a Convolutional Neural Network?

Answer 33

It’s a neural network where you have a convolutional layer.

Question 34

Why would you want to use a CNN?

Answer 34

It’s able to find local structure, and interactions between nearby components.

Question 35

What is “pooling”?

Answer 35

It’s where you combine the outputs of a convolutional layer, with something like average or L2 norm, to zoom out on the convolution region.

Question 36

What does RNN stand for?

Answer 36

Recurrent Neural Network

Question 37

What is unique about a Recurrent Neural Network?

Answer 37

It captures information about a sequence of values, by adding a loop to the hidden unit.

Question 38

What is the “transition function” in the context of an RNN?

Answer 38

The transition function is the term for the weights, because they are constant and don’t vary with time.

Question 39

What is the “unrolled network”?

Answer 39

It’s an RNN where you’ve expanded the loop into a bunch of iterations side by side – so the network at time 1, then feeding into itself at time 2, and then time 3. It ends up looking like a ladder.

Question 40

What does LSTM stand for?

Answer 40

Long short-term memory

Question 41

What is unique about an LSTM?

Answer 41

It has memory units that hang onto information over some duration.

Question 42

What does GAN stand for?

Answer 42

Generative Adversarial Network

Question 43

What are the two parts of a GAN?

Answer 43

First, a Generator creates fake examples to try and fool the discriminator. Then the Discriminator learns to discern real examples from fake generated ones. And the two are learned together.

Question 44

What is an “adversarial example”?

Answer 44

It’s one that has been minimally changed from a real example, but that causes the network to produce a very different prediction.

Question 45

What is “dataset augmentation”?

Answer 45

It’s when you add a bunch of new manually-concocted examples to your training set to try and help the model generalize.

Question 46

What are example of how people do dataset augmentation?

Answer 46

They might crop the data, or rotate it, or add noise, or erase random bits.