Neural network fundamental

Question 1

Neural network representation

Answer 1

Question 2

Why is deep learning taking off only now

Answer 2

• Amounts of data
• Faster computation (SPecialised GPU)
• Algorihtms
  
  • RELU activation function faster than sigmoid (sigmoid has smaller gradients for larger local fields)

Question 3

Describe Tanh() in terms of sigmoid

Answer 3

Tanh activation function is a shifted version of sigmoid where range is -1 to 1

Question 4

Tanh activation vs sigmoid activation

Answer 4

• Tanh is superior because the mean of the layer is closer to zero which makes learning for the next layer easier
• Only use sigmoid as activation for outputlayer when doing binary classification (1-0 values for output)
• A downside for both acivation functions is that if z (local field of the neuron) is very large, then the slop/gradient is very small, making gradient descent slow

Question 5

Softmax function

Answer 5

• Each output is between 0 to 1
• Output layers adds up to 1
• Can be viewed as a vector of probabilities
• Only used in outputlayer for multiclass classification
• combined with cross entropy loss

Question 6

Softmax activation function:

Visualize Simple neural net with 1 softmax layer

Answer 6

• Multiclass logistic regression
• Linear decision boundaries separating classes

Question 7

Forward propagation

Answer 7

• For vectorized we can feed the whole dataset X
• Each collumn result in activations layer represents an data input

Question 8

Vectorized forward propagation

Answer 8

Question 9

Intution in deep learning

Answer 9

• Earlier layers detecting simpler functions
• then composing these function to form more complex patterns (i.e eyes, mouth)
• Same applies for other domains not just images
  
  • i.e in sound low level functions can be audio/tone, later layers could represent complex forms such as phrases

Question 10

Circuit theory and deep learning

Answer 10

• Computing xor with one hidden layer is exponetionally large (right network)
• While computing with several layers is uses less neurons (left image)

Question 11

Optimization based learning

Answer 11

Question 12

Empirical risk

Answer 12

• An approximation of the true expected loss based on training data.
• Training data consists of a finite set of samples from the true distribution p(x,y) it does not caputre the distribution fully

Question 13

How to make neural netowork generalize better on unseen data?

Answer 13

• Adjust the loss to add regularization terms
• Split data into training, validation, and test

Question 14

Optimization based learning

Answer 14

Question 15

Empirical risk vs generlization

Answer 15

Question 16

Using training data, to formulate loss

Answer 16

Question 17

Local optima in neural networks

Answer 17

Question 18

Problem of plateau

Answer 18

Question 19

Maximum liklihood estimation

Answer 19

• a method of estimating the parameters of a probability distribution by maximizing a likelihood function, so that under the assumed statistical model the observed data is most probable.

Question 20

One hot encoding

Answer 20

Question 21

When should we use cross-entropy loss function ?

Answer 21

Classification when we use softmax function on the output layer

Question 22

One hot encoding and the cross entropy loss

Answer 22

Question 23

Broadcasting

Answer 23

• describes how numpy treats arrays with different shapes during arithmetic operations. the smaller array is “broadcast” across the larger array so that they have compatible shape
• i.e C = A + b where C_{ij}= A_{ij} + b_j
  
  • In other words, the vector b is added to each row of the matrix.
  • this shorthand eliminates the need to define a matrix B copied into each row before doing the addition

Question 24

Batch vs minbatch

Answer 24

Batch means that you use all your data to compute the gradient during one iteration. Mini-batch means you only take a subset of all your data during one iteration.

Question 25

Epoch

Answer 25

* Single pass through the whole training set * We can divide a dataset of 2000 examples into batches of 500 then it will take 4 iterations to complete 1 epoch. Where Batch Size is 500 and Iterations is 4, for 1 complete epoch. * The number of batches is equal to number of iterations for one epoch.

Question 26

How does the training progress look like for Batch training and min-batch training

Answer 26

* The loss should always go down for batch training * If it ever goes up then something is wrong (i.e learning rate is too high) * Minbatch training is slightly noiser, but the overall trend is that the loss is reducing with amount of iterations

Question 27

*Stochastic gradient descent*

Answer 27

* Each data sample is its own batch (size of batch = 1) * Very noisy, but * Won't ever completley converge to the minimum, but jump around it * Loosing speed up from vectorization

Question 28

*Minbatch-gradient descent*

Answer 28

* Train on a subset of data (minbatch) rather than the whole training data * Benefites (Works best in practice as to batch training or stochastic) * See progress for gradient descent without processing the entire dataset * Converge quicker because each gradient step is smaller * Less likley to converge to local minima as opposed to batch * Takes less main memory * Randomize the samples in each minibatch each epoch

Question 29

*Batch training*

Answer 29

* Use the whole dataset during during each step of gradient descent * Slower than minbatch or stochastic * However, more accurate and precise progress

Question 30

Visually explain * Batch training * Minbatch training * Stochastic gradient descent

Answer 30

Question 31

Gradient descent with *momentum*. Describe the "*momentum"*

Answer 31

* Compute an exponentially weighted average of the gradients * Each gradient descent step depend on previous steps On iteration *t* compute dW, db on the current minbatch v_dW = ßv_dW + (1-ß)dW v_db = ßv_db + (1-ß)dW W = W - Πv_dW , b = b - Πv_db * The second line approximates a gradient descent step, where V_dW is an approximation to the gradient. * The vector V_dW is an exponentially weighted average of the last gradients dW, which reduces the oscillations in dW Hyperparameters: learning rate Π, and exponentially weighted average ß ß = 0.9 Average over the last 10 gradients ß = 0.5 Average over the last 2 gradients

Question 32

RMS prop

Answer 32

* Exponentially weighted of the squares * Allows to select a larger learning rate

Question 33

Gradient descent with *Adam.* Describe *Adam*

Answer 33

* Adaptive moment estimation * Combines momentum and RMSprop * ß₁ * Called the first moment exponentially weighted averge * 0.9 * ß₂ * Second moment (exponentially weighted averag eof squares) * 0.99 * Σ = 10^-8 Used in order to not devide by zero

Question 34

Learning rate decay

Answer 34

* Initial steps of learning we can have larger *learning rate* * After a while, gradient descent will jump around a minimum, then we want to select a smaller learning rate * Where Π₀ initial learning rate Π = 1/(1+ decayRate\* epochsRun)Π₀ Π = 0.95^epochRun\* Π₀ (Exponentially decay) Π = Divide by 2 each epoch (Discrete staircase) * Decay rate becomes another hyper parameter * Manual decay * Manually decrease learning rate after gradient descent has been running * Only works if training small amount of models

Question 35

Train/dev/test set

Answer 35

* Split data into different parts train/dev/test * This split is used to tune the hyper parameters (i.e learning rate) of a model * Hyperparameters in a neural network * #Layers * #Neurons * Activations function * Learning rate * Momentum * If your data is small than use classical proportions 70/20/10 * If your data is big, then use modern (big data era) proportions 98/1/1

Question 36

Bias/variance

Answer 36

* **Bias:** Performance on the training data compared to optimal performance * **Variance:** difference between loss on training and validation data * High variance * Overfitting * Example (assume human error ≈ 0) * Train error: 1% * Dev error: 11% * High bias * Underfitting * Example (assume human error ≈ 0) * Train error: 15% * Dev error:16% * Low bias/low variance * Best case * Train error matches dev error * Example (assume human error ≈ 0) * Train error: 0.5% * Dev error: 1% * High bias/High variance * Worst case * Model very has high parameters and is flexible but mistrains on samples * Example (assume human error ≈ 0 in cat/dog pictures) * Train error: 15% * Dev error: 30%

Question 37

Basic recipe for NN

Answer 37

* If High bias issue, then getting more data is not going to help * Less bias/variance tradeoff issue in NN

Question 38

Regularization

Answer 38

* Tuned as a hyper parameter * L1 - regularization * Weight vector *w* becomes sparse (contain many zeros) * L2 - regularization * Euclidearn norm

Question 39

Show how regularization "decays the weight"

Answer 39

Question 40

Normalizing data

Answer 40

* Subtract each sample by zero mu = 1/m\*( Σ^m_i=1 x_i ) x: = x - mu * Divide each sample by mu = 1/m\*( Σ^m_i=1 x_i \*\*2 ) (\*\*2 means elementwise squaring) Get mu and sigma from training and use these same values to also normalize test data. * Why normalize * The scale in each input feature might differ drastically * The difference scaling in each features lead that gradient descent steps oscillate * More cerical controus leads to that gradient descent needs less steps to converge

Question 41

*Batch normalization*

Answer 41

* Makes hyperparameter search easier * Makes network much more robust to choice of paramters * Much bigger range of hyperparameters that work well * Apply same process of normalization of inputs for all z values (inputs to neurons) at each layer * Later layers are more robust to changes in earlier layers * Allows each layer to learn independently from other layers * Reduces covariate shift * If distribution of X changes then have to re-learn training algorithm, This is true even if the ground true mapping X=\>y remains unchanged * Batch norm at test time * Come up with a separate estimate for mean and deviation **during training** and not on the test set * i.e exponentially weighted average or any other average methods on minbatches

Question 42

Explain informally how regularization reduces variance

Answer 42

Question 43

Give reasons why Image classification is so hard

Answer 43

* Hidden parts * Deformable objects * Viewing objects from an werid angle

Question 44

Define *Gradeint descent,* and show how to update the weights

Answer 44

* Computes the gradient of the loss and updates the weights in the opposite direction of the gradient.

Question 45

Gradeint descent. How does each step change cost function

Answer 45

Question 46

Forward mode differentiation

Answer 46

Question 47

Backward mode- differentiation

Answer 47

* Also known as backprop

Question 48

Answer 48

Question 49

Answer 49

Question 50

Answer 50

Question 51

Exponentially weighted average

Answer 51

.

Question 52

Bias correction

Answer 52

* Associated with exponentially weighted average

Question 53

*Backpropagation*

Answer 53

In back-propagation, the gradient of the loss is computed with respect to all variables in the function such that parameters can be updated using gradient descent.

Question 54

Answer 54

* The first is not correct, the number of paths would grow exponentially * The second one is correct

Question 55

Adam optimization

Answer 55

First and fourth is correct

Question 56

Does the runtime for performing *Backpropagation* depend on the amount of samples

Answer 56

Yes. We have to take the gradeint of each term separately

Question 57

Answer 57

Quickly