Module 01 - Fundamentals of Neural Networks

Question 1

Neural Networks (Part 1): Model Representation

What is this an example of?

Answer 1

A nonlinear classification problem

Question 2

Neural Networks (Part 1): Model Representation

Why did ANNs become popular again? (2)

Answer 2

• Better computer architecture (GPUs, parallelism)
• More data

Question 3

Neural Networks (Part 1): Model Representation

What’s the difference between the old and new views of ANNs?

Answer 3

Previously: function approximators.
Now: Interesting intermediate representations.

Question 4

Neural Networks (Part 1): Model Representation

What do neurons consist of? (ADS)

Answer 4

A neuron consists of:
- Axon (Single long fiber, output)
- Dendrites (Fibers, inputs)
- Soma (Cell body)

Question 5

Neural Networks (Part 1): Model Representation

Where is information processed/stored in the brain?

Answer 5

Simultaneously throughout the whole network instead of specific locations.
(Though parts of the brain specialize.)

Question 6

Neural Networks (Part 1): Model Representation

Where is the nucleus located? (See image)

Answer 6

Question 7

Neural Networks (Part 1): Model Representation

Where are the dendrites located? (See image)

Answer 7

Question 8

Neural Networks (Part 1): Model Representation

Where is the cell body (soma) located? (See image)

Answer 8

Question 9

Neural Networks (Part 1): Model Representation

Where is the node of ranvier located? (See image)

Answer 9

Question 10

Neural Networks (Part 1): Model Representation

Where is the axon located? (See image)

Answer 10

Question 11

Neural Networks (Part 1): Model Representation

Where is the myelin sheath located? (See image)

Answer 11

Question 12

Neural Networks (Part 1): Model Representation

Where is the Schwann cell located? (See image)

Answer 12

Question 13

Neural Networks (Part 1): Model Representation

Where is the axon terminal located? (See image)

Answer 13

Question 14

Neural Networks (Part 1): Model Representation

What is the purpose of the dendrites?

Answer 14

They are the input channels to the neuron.

Question 15

Neural Networks (Part 1): Model Representation

What components are the input channels to the neuron?

Answer 15

The dendrites.

Question 16

Neural Networks (Part 1): Model Representation

What is the purpose of the axon?

Answer 16

It’s the output of the neuron.

Question 17

Neural Networks (Part 1): Model Representation

What is the output channel of the neuron called?

Answer 17

The axon.

Question 18

Neural Networks (Part 1): Model Representation

What is an activation function?

Answer 18

Any function applied to the outputs of a neural network, e.g. sigmoid or ReLU.

Question 19

Neural Networks (Part 1): Model Representation

What is the “step function” activation function?

Answer 19

Question 20

Neural Networks (Part 1): Model Representation

What is the “sign function” activation function?

Answer 20

Question 21

Neural Networks (Part 1): Model Representation

What is the “sigmoid function” activation function?

Answer 21

Question 22

Neural Networks (Part 1): Model Representation

What is the “linear function” activation function?

Answer 22

Question 23

Neural Networks (Part 1): Model Representation

What is a requirement for activation functions?

Answer 23

They have to be differentiable.

Question 24

Neural Networks (Part 1): Model Representation

What is a logistic unit?

Answer 24

A neuron with a sigmoid function (e.g. logistic function) applied to the outputs.

Question 25

# Neural Networks (Part 1): Model Representation What is multiclass classification?

Answer 25

A neural network predicting multiple classes as once, e.g. cat or dog or rabbit.

Question 26

# Neural Networks (Part 1): Model Representation What's another name for one-vs-all classification?

Answer 26

Multiclass classification

Question 27

# Neural Networks (Part 1): Model Representation What is Hebbian learning?

Answer 27

If 2 units are both active (firing), the weights between them should increase

Question 28

# Neural Networks (Part 1): Model Representation What's the name for the "Neurons that fire together wire together" rule.

Answer 28

Hebbian learning.

Question 29

# Neural Networks (Part 1): Model Representation What is a feedforward neural network?

Answer 29

A NN with no cycles (Connections back).

Question 30

# Neural Networks (Part 1): Model Representation What do we call an NN with cycles?

Answer 30

A recurrent neural network.

Question 31

# Neural Networks (Part 1): Model Representation What is a recurrent neural network?

Answer 31

A NN with cycles?

Question 32

# Neural Networks (Part 2): Learning In Hebbian learning, are learning rules local or global?

Answer 32

Local

Question 33

# Neural Networks (Part 2): Learning What's the idea behind Hebbian learning in NNs?

Answer 33

Change weights based on correlation of connected neurons

Question 34

# Neural Networks (Part 2): Learning When does Hebbian learning work best?

Answer 34

Works best when relevance of inputs to outputs is independent

Question 35

# Neural Networks (Part 2): Learning What's the problem with Hebbian learning and weights?

Answer 35

Simple Hebb rule grows weights unbounded

Question 36

# Neural Networks (Part 2): Learning What's the perceptron learning rule?

Answer 36

New weights are old weights plus a portion of the error.

Question 37

# Neural Networks (Part 2): Learning What's the Widrow-Hoff Rule?

Answer 37

The Widrow-Hoff rule aims to minimize the mean square difference between the predicted (expected) and the actual (observed) data or response

Question 38

# Neural Networks (Part 2): Learning How do you calculate the gradient of a multi-variable function? (See image)

Answer 38

Question 39

# Neural Networks (Part 2): Learning What is gradient descent?

Answer 39

Following the gradient of a function in an attempt to reach the global minima.

Question 40

# Neural Networks (Part 2): Learning Under what situations will gradient descent reach the global minimum of a function?

Answer 40

When the function is convex.

Question 41

# Neural Networks (Part 2): Learning What is the gradient descent update rule? (Formula)

Answer 41

Question 42

# Neural Networks (Part 2): Learning What is the squared error formula?

Answer 42

Question 43

# Neural Networks (Part 2): Learning What are the parameters we want to train with gradient descent?

Answer 43

Weights (+ bias) and thresholds.

Question 44

# Neural Networks (Part 2): Learning What is the requirements for using the backpropagation algorithm?

Answer 44

All activation functions have to be differentiable, as well as the error function.

Question 45

# Neural Networks (Part 2): Learning How do you calculate the derivative of z with respect to x? (See image)

Answer 45

Question 46

# Neural Networks (Part 2): Learning How do you calculate the derivative of z with respect to x, when the path to z branches? (See image)

Answer 46

Question 47

# Neural Networks (Part 2): Learning What is gradient checking?

Answer 47

Numerically approximating the gradients.

Question 48

# Neural Networks (Part 2): Learning How do you use gradient checking?

Answer 48

It's used to check if backprop is properly implemented.

Question 49

# Neural Networks (Part 2): Learning What technique do we use to check if backprop is properly implemented?

Answer 49

Gradient checking (numerical technique).

Question 50

# Neural Networks (Part 2): Learning How are the parameters of a NN initialized?

Answer 50

They are initialised with random values

Question 51

# Neural Networks (Part 2): Learning Why is "Symmetry breaking" important?

Answer 51

When NN models are initialized with similar parameters, gradient descent doesn't know which one to update. Random parameters

Question 52

# Neural Networks (Part 2): Learning What is deep learning?

Answer 52

The study of neural networks with 3+ layers.

Question 53

# Neural Networks (Part 2): Learning What is SGD with mini-batches?

Answer 53

Stochastic gradient descent with more than one training example at a time.

Question 54

# Neural Networks (Part 2): Learning What is SGD short for?

Answer 54

Stochastic gradient descent.

Question 55

# Neural Networks (Part 2): Learning What is weight decay

Answer 55

Decaying the weight by multiplying by a constant c after every epoch.

Question 56

# Neural Networks (Part 2): Learning What is weight decay helpful for?

Answer 56

Reducing overfitting.

Question 57

# Neural Networks (Part 2): Learning What is weight decay similar to?

Answer 57

Adding a weight regularization terms to the error.