Chapter 8- Neural Networks

Question 1

what type of neural networks demonstrate above human level performance in chess and go?

Answer 1

convolutional neural networks

Question 2

what is alex net?

Answer 2

a convolutional neural network that outperformed other models in the imagenet challenge

Question 3

what is a spiking neural network?

Answer 3

it aims to mimic a biological neuron more closely

Question 4

give a neuron mathematically in sum notation y(x,y) =

Answer 4

f(sum: wx +b)

Question 5

give a neuron mathematically in matrix notation

Answer 5

f(W.X)

Question 6

give three different activation functions?

Answer 6

threshold, sigmoid, softmax

Question 7

what is the activation function used for logistic regression?

Answer 7

sigmoid

Question 8

give the sigmoid activation function y(X,W) = ?

Answer 8

1 / (1 + e^-z)

Question 9

when is the softmax activation function used?

Answer 9

when we have multiple, mutually exclusive classes

Question 10

softmax is an extension of…?

Answer 10

the logistic function

Question 11

give the equation for gradient descent, w_new = ?

Answer 11

w_old - lamda (dL/dw)

Question 12

give the equation for squared error loss, L = ?

Answer 12

0.5(y-t)^2

Question 13

what is backpropagation?

Answer 13

the application of the chain rule for neural networks

Question 14

what are the two stopping criteria we use for neural networks?

Answer 14

maximum number of epochs

early stopping criteria

Question 15

what does the learning rate determine?

Answer 15

how large an adjustment we make to each weight at each iteration

Question 16

what neural network structure should be sufficient to approximate any function?

Answer 16

a multilayer perceptron with one hidden layer

Question 17

what is the advantage of adding more layers to a model, rather than more neurons?

Answer 17

increases flexibility with fewer free parameters

Question 18

what are the three approaches to establishing neural network architecture?

Answer 18

experimentation, heuristics, pre-trained models (transfer learning)

Question 19

when we add too many layers/neurons to a model, we risk…?

Answer 19

overfitting

Question 20

what is bias error?

Answer 20

error due to an erroneous assumption in the model

Question 21

what is variance error?

Answer 21

error due to the algorithm fitting to noise in the training data

Question 22

what kind of error decreases as we make a model more complex?

Answer 22

bias

Question 23

describe the idea behind a drop out scheme

Answer 23

begin with an overly complex model

during training, the output of any individual neuron is ignored with probability p

Question 24

what is the traditional error curve of simpler models

Answer 24

test error decreases up until the model is sufficiently complex and then increases

Question 25

what is double descent?

Answer 25

if we continue to increase the number of hidden layers, the test error decreases again

Question 26

what are the two problems that deep(er) neural networks face?

Answer 26

training time

vanishing gradient problem

Question 27

what is the vanishing gradient problem

Answer 27

the weight update in the early layers can be extremely close to zero

Question 28

what are the two ways to fix the vanishing gradient problem?

Answer 28

relu, rectified linear unit activation function

feed the output of a neuron directly into a later stage of the network

Question 29

describe the relu activation function

Answer 29

all negative values for y are set to 0, has a gradient of 1 for positive values for y

Question 30

what is the input to a cnn?

Answer 30

raw 2d image

Question 31

how do we represent each layer of a cnn?

Answer 31

rectangle

Question 32

what is a fully connected layer?

Answer 32

each neuron in the current layer is connected to all neurons in the next layer

Question 33

what do the different layers of the cnn learn, how is it different to a standard mlp? (hint: features)

Answer 33

early layers learn the features that are used for classification, rather than them needing to be pre-selected

Question 34

give a kernel for horizontal lines

Answer 34

1 1 1
0 0 0
-1 -1 -1

Question 35

give a kernel for vertical lines

Answer 35

1 0 -1
1 0 -1
1 0 -1

Question 36

what is padding?

Answer 36

ensure the size of the output remains the same size by extending the original image in a non informative way

Question 37

what is a pooling operation?

Answer 37

select the (max/min/average) value in a local area

Question 38

what is the effect of pooling, why do we do it?

Answer 38

makes the model more robust to variations in position

Question 39

what is the flatten layer?

Answer 39

converts the 2d map into a 1d array of features

Question 40

what is a recurrent neural network

Answer 40

include loops in the hidden layer neurons which allow the network to use historical information

Question 41

on what type of data are recurrent neural networks useful?

Answer 41

time series

sequence

Question 42

what is a generative adversarial network?

Answer 42

generates data that the discriminator thinks is from the training set.

Question 43

what are the two parts of a generative adversarial network?

Answer 43

generator and a discriminator

Question 44

relu vs sigmoid

Answer 44

relu is faster and solves the vanishing gradient problem

Question 45

cons of neural networks (2)

Answer 45

does not take into account spatial or temporal information

black box

Question 46

what is a capsule neural network

Answer 46

we not only have a representation of the image, put also its pose