Week 2

Question 1

Defining characteristic of DNN

Answer 1

More than 1 hidden layer

Question 2

Advantages of SGD

Answer 2

Efficient for large sample
Implementable numerically
Can be ‘controlled’

Question 3

RNN

Answer 3

Recurrent Neural networks sequentially feed output back into network

Question 4

Connection from FNN to RNN

Answer 4

RNNs can be reduced to FNNs by UNFOLDING them

Question 5

First NN

Answer 5

McCulloch and Pitts

Question 6

Perceptron & problems of

Answer 6

Rosenblatt ‘58

Question 7

What started AI winter

Answer 7

‘69 Minsky showed XOR couldn’t be replicated by perceptron

Question 8

FULLY CONNECTED

Answer 8

If all entries of each L_i in NN are non zero

Question 9

Universal approximation property

Answer 9

Let g:R -> R be a measurable function such that:
a) g is not a polynomial function

Question 10

Define FNN

Answer 10

Question 11

Differences between (hyper)params

Answer 11

Hyper:
Set by hand
Features

Params:
Chosen by machine (weights and biases)
Optimised by SGD

Question 12

Architecture of network

Answer 12

Hyperparameters and Activatuon Functions (things chosen by you)

Question 13

Dense layer

Answer 13

Entire layer is connected (non zero)

Question 14

Number of parameters that characterise N

Answer 14

Question 15

Adding units

Answer 15

Question 16

Continuity and differentiability of NN

Answer 16

Bottom line:
If every activation function is continuous, then so to is the NN

The NN is overall as differentiably continuous as the LEAST differentiable activation function

Question 17

One dimensional activation functions (entire table)

Answer 17

Question 18

Dead ReLU problem & solution

Answer 18

A layer of ReLU activations receives only negative values -> producing constant output

This can freeze gradient based algorithms

Therefore use leaky ReLU or Parametric ReLU (or ELU)

Usually 0 < α < 1

Question 19

Multi dimensional activation

Answer 19

Question 20

When is identity function useful

Answer 20

Output layer

Question 21

Limitation of Heaviside

Answer 21

As is non continuous, can’t be used in gradient based algorithms

Question 22

Saturating activation functions

Answer 22

Output is bounded

Sigmoid, tanh

Question 23

Continually differentiable counterpart of ReLU

Answer 23

Soft plus

Question 24

Boltzmann dist

Answer 24

From stat physics

Analogous to Multinomial logistic regression
Which is Standard activation function in image recognition

Question 25

Motivation for max out

Answer 25

Several, simple, convex non linear functions can be expressed as maxima of affine functions eg

ReLU(x) = max{0, x}
|x| = max{-x, x}

Question 26

Sup Norm

Answer 26

Question 27

L^p norm

Answer 27

Question 28

Def universal approximation property

Answer 28

Question 29

Limitation of UAP

Answer 29

Result is non constructive:
It does not tell what the approximating NNs f and h look like, just that they exist

Also it is non quantitative:
It doesn’t tell how many hidden units are required to create these networks