Lecture 15 - Neural Networks Part 3

Question 1

What is the generalised delta rule?

show the formula?

Answer 1

A rule that can be used to modify the weights to minimize |z-y| regardless of the order of the output function

σx_j=(z-y)(dy/dw_j)

Question 2

What is the formula of the logistic function?

What are its properties?

Answer 2

y = 1 / ( 1 + e^{-wbar xbar})

Curve has a sigmoid shape

Differentiable

Monotonically increasing (i.e. higher x is always higher y)

Tends to 1 as input tends to +inf

Tends to 0 as input tends to -inf

Equal to 0.5 when input equal to 0

Question 3

What is the derivative of the logistic function?

Answer 3

y(1-y)

Question 4

Why is training hidden units difficult?

Answer 4

No idea what their expected outputs should be

Question 5

What is the solution to the difficulty of training hidden units?

Answer 5

Devise a way of making plausible guesses at what the outputs should be

Question 6

To use the generalised delta rule, the output function must be ____________

Answer 6

Differentiable

Question 7

How are hidden units trained?

Answer 7

Feeding back the error from the output layer

-> the estimated error of a hidden unit is the weighted sum of the errors of the output units

Question 8

Passing back a weighted error to train a hidden unit is known as ?

Answer 8

back propagation

Question 9

What is the issue with large values of a in back propagation?

Answer 9

May make it impossible to find a minimum error value

Question 10

What is the issue with a small value of a in back propagation?

Answer 10

Prolong the gradient descent and increase the chance of finding a local (rather than global) minimum

Question 11

What are two ways of determining when to stop training using back propagation?

Answer 11

Keep going until error falls below a given threshold

Keep going until average error change is small

Question 12

How can we reduce overfitting in back propagation networks?

Answer 12

Decrease number of hidden units

Add weight decay (each iteration all weights decrease slightly)

Use large number of high quality training samples

Question 13

What is the problem with decreasing the number of hidden units and adding weight decay to combat overfitting in back propagation networks?

Answer 13

Restrict the complexity of the output function

Question 14

What is cross validation, as it relates to overfitting?

Answer 14

Split training data into two sets: one to modify weights and one to test the weights

Test the test data after each weight change

Use the weights that gave the best result

Question 15

For what purpose are back propagation networks particularly useful?

Answer 15

Approximation, regression, prediction, classification