Neural Network Basics Flashcards
Describe the Step Function, is it suitable for NN training?
1 if x>0, 0 otherwise. It helps to introduce nonlinearities
No, its not suitable for NN training
Describe the ReLU function, is it commonly used? if so, why?
relu(x) = max (0, x)
Yes, most commonly in modern architectures, because its very easy to compute, yet offers enough capability to build complex models
Describe the Sigmoid/Logistic Function
Where is it most important nowdays?
- Output in a specific range (0 & 1)
sigmoid(x) = 1/(1+e^x) - Nowadays mostly important in output neurons
Describe the tanh(x) function and its output
tanh(x) = 2 · sigmoid(2 · x) – 1
output is between -1 and 1
Describe the type of layers in a multilayer perceptron
Types of layers
– Input layer: Contains the inputs (e.g.,
pixel values of an image)
– Hidden layer: Internal layer of the network
– Output layer: Produces the outputs (e.g., classification probabilities)
How is the calculation performed in a MLP (multilayer perceptron)?
(neuron*weight) + bias
What’s inferencing?
Is using a model after training for new predictions
Whats the difference Metric vs. loss?
– Metric: What we (as humans) actually care about
– Loss: Drive automated learning; compromise between metric and a differentiable function
What’s the central loop in training all deep learning models? (Draw it & explain it)
- Init: Initialize the weights in some way, e.g., randomly
– Predict: Predict outputs with the current weights
– Loss: Compute how good the model performs
– Gradient: For each weight, how does changing it slightly affect the loss?
– Step: Adjust weights according to gradient
– Repeat until some stop condition is met
why gradients and the learning rate are important?
Gradients provide the direction for minimizing the loss function, guiding the parameter updates. The learning rate controls the step size of these updates, balancing the speed and stability of the learning process. Choosing appropriate values for these components is crucial for successful neural network training, as they directly impact the model’s ability to learn and generalize from data.
Which statement is true about the origins of neural networks?
- Neural networks were inspired by evolution. The training procedure (also called “fitting”) is a close resemblance of the “survival of the fittest”, where the best found solution improves over time due to evolutionary pressure.
- Neural networks were inspired by the brain and they closely model the neurons and their connections in the brain.
- Neural networks weren’t inspired by any biological analogy, the naming is rather a coincidence.
- Neural networks were inspired by biological neurons and their connections, but they remain a simplification of their biological counterparts.
Neural networks were inspired by biological neurons and their connections, but they remain a simplification of their biological counterparts.
You want to implement a neural network with the following specification:
- 5 input values, 10 output values
- one hidden layer of size 20, then a hidden layer of size 10
The network should first use a ReLU activation function, then a sigmoid and then a tanh at the output.
Provide the PyTorch code to implement this network
nn.Sequential(
nn.Linear(5, 20),
nn.ReLU(),
nn.Linear(20, 10),
nn.Sigmoid(),
nn.Linear(10, 10),
nn.Tanh()
)
You are given the following neural network defined in PyTorch:
nn.Sequential(
nn.Linear(9,13), nn.ReLU(), nn.Linear(13,3), nn.Sigmoid()
)
What is the total number of parameters created by this model?
(9x13) + 13 +(13*3)+ 3 = 172
Which statements are correct about metrics and loss functions? (Multiple Choice)
1. The metric is used in the training process to compute the loss function.
2. The loss function is the gradient of the parameters.
3. The loss function is generally the same as the metric.
4. The metric is used in the training process to give human feedback about model performance.
5. The loss function is usually similar to a metric, but it needs to be differentiable.
4 & 5
Who first proposed the concept of artificial neurons, then called perceptrons?
a) Hinton and Salakhutdinov
b) McCulloch and Pitts
c) LeCun and Bengio
d) Hochreiter and Schmidhuber
Answer: b) McCulloch and Pitts
