Neural Network Basics Flashcards
Linear function for neural network?
Signals from the input are multiplied by weights and summed up, then bias is added. Each parameter can be trained to yield the desired results
What is activation function?
Function that calculates the output of the node based on its individual inputs and their weights
What is step function?
1 for x>0, 0 otherwise (looks like a ladder). Not well suited for NN training
What is ReLU?
Rectified linear unit. relu(x) = max (0, x) - starts from 0 45 degrees, goes endlessly. Very easy to compute, yet offers enough capability to build complex models.
Algorithm of NN training
Initiate parameters (randomly) - predict on training data - compute loss - compute gradient - update parameters - repeat
What is sigmoid?
1 / 1+ e^-x, logistic function, ensures output is between 0 and 1
What is tanh?
Scaled and shifted sigmoid: 2 x sigmoid (2x)-1. Output is between -1 and 1. Used to be popular in place of ReLU
What is MLP?
Multi-Layer Perception: arrangement of neurons into multiple layers
What is inferincing?
Using a model after training to make new predictions. Inputs -> Model -> Results
What is SGD (Stochastic Gradient Decsent) ?
Optimization algorithm to minimize the loss function and adjust the model parameters (e.g., weights and biases) during training.
Explain the difference between a metric and a loss
Metric: What we (as humans) actually care about, changes if actually one of the predictions changes when we change the weights
Loss: Drive automated learning; compromise between metric and a differentiable function.
loss function reacts to small changes in predicted score, gives finegrained performance assessment
First code line for simple neural network
nn.Sequential(
Layer between activation functions
nn.Linear (input, output)
Why use activation function?
Adding Non-linearity, Helping the Network Decide
What is learning rate?
Small number that controls how much a neural network adjusts its parameters (like weights and biases) during training.
