NN FINAL

Question 1

Hidden layers are needed if

Answer 1

the data must be separated using a non-linear boundary

Question 2

Major difference between ANN and Perceptron

Answer 2

the inclusion of hidden layers

Question 3

Universal Approximation Theorem for Neural Networks

Answer 3

A FFNN with a single hidden layer containing an arbitrary number of neurons can approximate any continuous function

Question 4

Universal Approximation theorem also proved for

Answer 4

arbitrary number of hidden layers, each containing a limited number of neurons

Question 5

Hidden layers can represent

Answer 5

arbitrary complex decision boundaries

Question 6

Deep neural network meaning

Answer 6

referring to the depth of the hidden layers. Typically more than 3-4 hidden layers

Question 7

FFNN

Answer 7

feed information forwards

Question 8

Back prop

Answer 8

errors are propagated backwards to correct the weights

Question 9

Downstream

Answer 9

towards the right

Question 10

upstream

Answer 10

towards the left

Question 11

FFNN used for

Answer 11

General neural networks, classification, regression

Question 12

Convolutional Neural Networks

Answer 12

Excel at image recognition

Question 13

Recurrent Neural Networks

Answer 13

Excel at language tasks, and predicting next word

Question 14

Long short-term memory networks

Answer 14

Like RNN, but for tasks that require longer context

Question 15

Generative-Adverserial networks

Answer 15

Generative neural network is trained on generating something
Adverserial network is then trained on classifying what was generated as whether it was accurate or not

Question 16

Hidden nodes learn

Answer 16

latent representation (features useful for class boundaries)

Question 17

First hidden layer captures

Answer 17

simpler features (since it receives the predictors as input)

Question 18

Subsequent hidden layers hone into

Answer 18

specific patterns of the data to extract features

Question 19

What does a neuron do?

Answer 19

Exactly same thing we saw perceptron doing input where w transpose x happens, output part is where activation function is run

Question 20

Activation function is important, as it provides

Answer 20

non-linearity to an ANN and allows it to create non-linear class boundaries

Question 21

How to choose an activation function at output layer?

Answer 21

Match the activation function at the output layer based on the type of prediction problem

Question 22

output Activation function for regression

Answer 22

Linear activation function

Question 23

output Activation function for Binary classification

Answer 23

sigmoid/Logistic activation function

Question 24

output Activation function for Multiclassification

Answer 24

Softmax activation function

Question 25

How to choose an activation function at hidden layers

Answer 25

Start with relu activation function and move to others if results are sub-optimal

Question 26

What does it mean that NN is learning

Answer 26

Updating its weights

Question 27

What should we initialize the weight vector with?

Answer 27

Random initialization w = N(0, o^2), normal distribution with mean of 0 and standard deviation of sigma squared

Question 28

How to choose sigma squared?

Answer 28

Xavier initialization He initialization

Question 29

Xavier initialization

Answer 29

2/((# of neurons in prev layer) + (number of neurons in next layer))

Question 30

He initialization

Answer 30

2/(# of neurons in previous layer)

Question 31

Result of backpropogation

Answer 31

A gradient vector of weights used for updating the weights to get to the minimum gradient

Question 32

A gradient descent algorithm does

Answer 32

iteratively goes through the training dataset, modifying weights during each pass (epoch) to minimize the cost

Question 33

Batch gradient descent

Answer 33

Calculate error for each observation and at the end of the training data calculate average error and update w

Question 34

Stochastic gradient descent

Answer 34

Calculate error after each observation and update w

Question 35

Mini-batch gradient descent

Answer 35

Split data into small batches, calculate error for each observation in a batch and at the end of the batch calculate average error and update w Preferred method

Question 36

Neural networks are almost always

Answer 36

over-parametrized, yet they perform well

Question 37

over-parameterization leads to

Answer 37

better learning

Question 38

How much training data do i need for my neural network

Answer 38

no good answer; 10x more observations for training than there are parameters in your neural network

Question 39

Gradient Descent Algorithm definition

Answer 39

The use of gradients to explore the minima of the error function

Question 40

Gradient descent relates

Answer 40

Error function minimization Minimizes the error function with respect to its weights

Question 41

Weight update formula

Answer 41

New weight = old weight - minus the learning rate multiplied by the gradient of the loss function with respect to the weight