Quiz 1 - Linear Classifiers, Gradient Descent, Neural Networks

Question 1

Feedforward Neural Network

Answer 1

Approximate a function using a mapping that finds optimal parameters for function approximation. The layers of a neural network are a Directed Acyclic Graph (DAG)

Question 2

Default recommendation for activation function of modern neural networks

Answer 2

rectified linear unit (ReLU)

Question 3

What is the major difference between neural networks and basic linear models?

Answer 3

Nonlinearity of a neural network causes lost functions to become convex. As a result, nn usually train by iterative, gradient-based optimizers that minimize cost (rather than get cost to 0).

For stocastic gradient descent of nonconvex loss functions, there is no guarantee of convergence and sensitivity to initial parameter values.

Question 4

What is regularization

Answer 4

Modification we make to a learning algorithm that is intended to reduce its generalization error but not its training error.

Example: weight decay to prevent weight parameters from getting too large (and potentially overfitting to train data).

Question 5

Neural Network cost function (most common)

Answer 5

NN are trained on maximum likelihood and the cost function is the negative log-likelihood.

Question 6

Negative log-likelihood cost function

Answer 6

The cross-entropy between the training data and the model distribution (measure of difference between two probability distributions)

Question 7

___ and ___ often lead to poor results when used with gradient-based optimization

Answer 7

1) mean squared error
2) mean absolute error

Question 8

What is the purpose of the sigmoid output

Answer 8

Sigmoid activation function converts the output to a probability (0,1) while ensuring that there is a strong gradient for wrong answers.

Question 9

Many objective functions other than log-likelihood do not work as well with the softmax function. Why?

Answer 9

Objective functions that do not use a log to undo the exponent of the softmax fail to learn when the argument to the exponent becomes very negative, causing the gradient to vanish.

Question 10

Common hidden layer unit

Answer 10

Relu

• Differentiable (except at 0)
• outputs zero across half its domain

Question 11

Different types of ReLU functions

Answer 11

• ReLU with non-zero slope for z_i < 0
  
  • Absolute Value Rectification
    
    • slope is non-zero and set to -1
    • used for object recognition from images
  • Leakly ReLU
    
    • slope is non-zero and small, around 0.01
  • parametric ReLI
    
    • slope of ReLU is a learnable parameter

Question 12

What is the problem with using the sigmoid function in a hidden layer?

Answer 12

Sigmoidal units saturate across most of their domain.

When z is strongly positive, they saturate to a high value and when z is strongly negative, they saturate to a low value.

The sigmoidal function is only strongly sensitive near z = 0.

Question 13

What is typically true of deeper neural networks?

Answer 13

They are often able to use far fewer units per layer and far fewer parameters

Question 14

universal approximation theorem

Answer 14

a feedforward network with a linear output layer and at least one hidden layer with any squashing activation function can approximate any Borel measurable function from one finite-dimensional space to another with any desired nonzero amount of error, provided that the network is given enough hidden units.

Question 15

Forward Propagation

Answer 15

The neural network input provides information that propagates up the hidden units at each layer and finally produces the output y’

Question 16

back-propagation “backprop”

Answer 16

Allows information from the cost to flow backward through the network in order to compute the gradient

Question 17

Scalar derivative rules

(fill out the table)

Answer 17

Question 18

What is the gradient of a function f(x,y)?

Answer 18

The vector of its partial derivatives

[df(x,y)/dx, df(x,y)/dy]

Question 19

What is the Jacobian of two functions f(x,y) and g(x,y)?

Answer 19

The matrix of the partial derivatives (the gradients for each function are rows)

Question 20

for neural networks, what is the dot product of vector:

w . x

Answer 20

The dot product of w . x is the summation of the element-wise multiplication of the elements

(ie. w . x = w^Tx)

Question 21

What does a linear classifier consist of?

Answer 21

• an input
• a function of the input
• a loss function

* One function decomposed into building blocks

Question 22

What modulates the output of the “neuron”

Answer 22

a non-linear function (ie. sigmoid) modulates the neuron output to acceptable values.

Question 23

The Linear Algebra View of a 2-layer neural network

Answer 23

The second layer (hidden) between the input and output layers corresponds to adding another weight matrix to the network

f(x, W₁, W₂) = sig(W₂ sig(W₁x))

Question 24

Two layered networks can represent any _____ function

Answer 24

continous

Question 25

Three layered neural networks can represent any ____ functions

Answer 25

(leave blank)

theoretically 3 layer neural networks can represent any function although in practice it may require exponentially large number of nodes

Question 26

What is the general framework for NN Computation graphs

Answer 26

• Directed Acyclic graphs (DAG)
• Modules in graph must be differentiable for gradient descent
• Training algorithm processes the graph one module at a time
• compositionality is achieved by this process

Question 27

Computation Graph example for NN

-log (1 / (1 + e^-wx) )

Answer 27

Question 28

Overview of backpropagation

Answer 28

• Calculate the current model’s outputs (outputs from hidden layer l - 1)
  
  • aka forward pass
• Calculate the gradients for each model
  
  • aka backward pass

Question 29

Backward pass algorithm “backpropagation”

Answer 29

• start at a loss function to calculate gradients
  
  • calculate gradients of the loss w.r.t. module’s parameters
• progress backwards through modules
  
  • given gradient of the output, compute gradient of the input and pass it back
• end in the input layer where no gradient needs to be computed.

Question 30

Backpropagation is the application of ___ to a ___ via the ___

Answer 30

1. gradient descent
2. computation graph
3. chain rule

Question 31

How do you compute the gradients of the loss (circled below)?

Answer 31

Question 32

reverse-mode automatic differentiation

Answer 32

• Given an ordering (ie. a DAG), iterate from the last module backwards, applying the chain rule, then store (for each node), its gradient outputs for efficient computation
  
  • forward pass: store activation
  • backwards pass: store the gradient outputs
    *

Question 33

Auto-Diff

Answer 33

A family of algorithms for implementing chain-rule on computation graphs

Question 34

What computation is performed for gradients from multiple paths?

Answer 34

Question 35

Patterns of Gradient Flow: Addition

Answer 35

Addition operation distributes gradients along all paths

Question 36

Patterns of Gradient Flow: Multiplication

Answer 36

Multiplication operation is a gradient switcher (multiplies it by the value of the other term)

Question 37

Patterns of Gradient Flow: Max Operator

Answer 37

Gradient flows along the path that was selected to be the max (which must be recorded in the forward pass)

Question 38

What is one of the most important aspects in deep neural networks that can cause learning to slow or stop if not done properly?

Answer 38

* the flow of gradients *

Question 39

What is forward- mode automatic differentiation

Answer 39

start from inputs and propagate gradients forward (no backwards pass)

*not common in deep learning because inputs are large (ie. images) and outputs (loss) are small

Question 40

Differentiable programming

Answer 40

• Computational graphs are not limited to mathematical functions
  
  • can have control flows (statements, loops)
  • backpropagate through algorithms
  • done dynamically so gradients are computed then nodes are added in repeat

Question 41

Derivative of sigmoid?

Answer 41

sigmoid(x)*(1 - sigmoid(x))

Question 42

derivative of cos(x)

Answer 42

-sin(x)

Question 43

derivative of tanh(x)

Answer 43

sec²(x)