Module 01 - Fundamentals of Neural Networks Flashcards
Neural Networks (Part 1): Model Representation
What is this an example of?
A nonlinear classification problem
Neural Networks (Part 1): Model Representation
Why did ANNs become popular again? (2)
- Better computer architecture (GPUs, parallelism)
- More data
Neural Networks (Part 1): Model Representation
What’s the difference between the old and new views of ANNs?
Previously: function approximators.
Now: Interesting intermediate representations.
Neural Networks (Part 1): Model Representation
What do neurons consist of? (ADS)
A neuron consists of:
- Axon (Single long fiber, output)
- Dendrites (Fibers, inputs)
- Soma (Cell body)
Neural Networks (Part 1): Model Representation
Where is information processed/stored in the brain?
Simultaneously throughout the whole network instead of specific locations.
(Though parts of the brain specialize.)
Neural Networks (Part 1): Model Representation
Where is the nucleus located? (See image)
Neural Networks (Part 1): Model Representation
Where are the dendrites located? (See image)
Neural Networks (Part 1): Model Representation
Where is the cell body (soma) located? (See image)
Neural Networks (Part 1): Model Representation
Where is the node of ranvier located? (See image)
Neural Networks (Part 1): Model Representation
Where is the axon located? (See image)
Neural Networks (Part 1): Model Representation
Where is the myelin sheath located? (See image)
Neural Networks (Part 1): Model Representation
Where is the Schwann cell located? (See image)
Neural Networks (Part 1): Model Representation
Where is the axon terminal located? (See image)
Neural Networks (Part 1): Model Representation
What is the purpose of the dendrites?
They are the input channels to the neuron.
Neural Networks (Part 1): Model Representation
What components are the input channels to the neuron?
The dendrites.
Neural Networks (Part 1): Model Representation
What is the purpose of the axon?
It’s the output of the neuron.
Neural Networks (Part 1): Model Representation
What is the output channel of the neuron called?
The axon.
Neural Networks (Part 1): Model Representation
What is an activation function?
Any function applied to the outputs of a neural network, e.g. sigmoid or ReLU.
Neural Networks (Part 1): Model Representation
What is the “step function” activation function?
Neural Networks (Part 1): Model Representation
What is the “sign function” activation function?
Neural Networks (Part 1): Model Representation
What is the “sigmoid function” activation function?
Neural Networks (Part 1): Model Representation
What is the “linear function” activation function?
Neural Networks (Part 1): Model Representation
What is a requirement for activation functions?
They have to be differentiable.
Neural Networks (Part 1): Model Representation
What is a logistic unit?
A neuron with a sigmoid function (e.g. logistic function) applied to the outputs.
Neural Networks (Part 1): Model Representation
What is multiclass classification?
A neural network predicting multiple classes as once, e.g. cat or dog or rabbit.
Neural Networks (Part 1): Model Representation
What’s another name for one-vs-all classification?
Multiclass classification
Neural Networks (Part 1): Model Representation
What is Hebbian learning?
If 2 units are both active (firing), the weights between them should increase
Neural Networks (Part 1): Model Representation
What’s the name for the “Neurons that fire together wire together” rule.
Hebbian learning.
Neural Networks (Part 1): Model Representation
What is a feedforward neural network?
A NN with no cycles (Connections back).
Neural Networks (Part 1): Model Representation
What do we call an NN with cycles?
A recurrent neural network.
Neural Networks (Part 1): Model Representation
What is a recurrent neural network?
A NN with cycles?
Neural Networks (Part 2): Learning
In Hebbian learning, are learning rules local or global?
Local
Neural Networks (Part 2): Learning
What’s the idea behind Hebbian learning in NNs?
Change weights based on correlation of connected neurons
Neural Networks (Part 2): Learning
When does Hebbian learning work best?
Works best when relevance of inputs to outputs is independent
Neural Networks (Part 2): Learning
What’s the problem with Hebbian learning and weights?
Simple Hebb rule grows weights unbounded
Neural Networks (Part 2): Learning
What’s the perceptron learning rule?
New weights are old weights plus a portion of the error.
Neural Networks (Part 2): Learning
What’s the Widrow-Hoff Rule?
The Widrow-Hoff rule aims to minimize the mean square difference between the predicted (expected) and the actual (observed) data or response
Neural Networks (Part 2): Learning
How do you calculate the gradient of a multi-variable function? (See image)
Neural Networks (Part 2): Learning
What is gradient descent?
Following the gradient of a function in an attempt to reach the global minima.
Neural Networks (Part 2): Learning
Under what situations will gradient descent reach the global minimum of a function?
When the function is convex.
Neural Networks (Part 2): Learning
What is the gradient descent update rule? (Formula)
Neural Networks (Part 2): Learning
What is the squared error formula?
Neural Networks (Part 2): Learning
What are the parameters we want to train with gradient descent?
Weights (+ bias) and thresholds.
Neural Networks (Part 2): Learning
What is the requirements for using the backpropagation algorithm?
All activation functions have to be differentiable, as well as the error function.
Neural Networks (Part 2): Learning
How do you calculate the derivative of z with respect to x? (See image)
Neural Networks (Part 2): Learning
How do you calculate the derivative of z with respect to x, when the path to z branches? (See image)
Neural Networks (Part 2): Learning
What is gradient checking?
Numerically approximating the gradients.
Neural Networks (Part 2): Learning
How do you use gradient checking?
It’s used to check if backprop is properly implemented.
Neural Networks (Part 2): Learning
What technique do we use to check if backprop is properly implemented?
Gradient checking (numerical technique).
Neural Networks (Part 2): Learning
How are the parameters of a NN initialized?
They are initialised with random values
Neural Networks (Part 2): Learning
Why is “Symmetry breaking” important?
When NN models are initialized with similar parameters, gradient descent doesn’t know which one to update.
Random parameters
Neural Networks (Part 2): Learning
What is deep learning?
The study of neural networks with 3+ layers.
Neural Networks (Part 2): Learning
What is SGD with mini-batches?
Stochastic gradient descent with more than one training example at a time.
Neural Networks (Part 2): Learning
What is SGD short for?
Stochastic gradient descent.
Neural Networks (Part 2): Learning
What is weight decay
Decaying the weight by multiplying by a constant c after every epoch.
Neural Networks (Part 2): Learning
What is weight decay helpful for?
Reducing overfitting.
Neural Networks (Part 2): Learning
What is weight decay similar to?
Adding a weight regularization terms to the error.
