Multi-Layer Neural Network

Question 1

What is the issue with a perceptron?

Answer 1

It cannot separate anything that is not linearly separable

Question 2

What is a multi-layer perceptron?

Answer 2

> Contains multiple neurons

> Contains multiple layers

• Input layer
• Hidden layer
• Output layer

Question 3

What is the simplest example of something that is not linearly separable?

Answer 3

XOR gate

Question 4

Theoretically, how many layers is required for a universal aproximator?

Answer 4

2 layers

Question 5

What is the issue with just 2 layers?

Answer 5

In theory only 2 layers are enough to aproximate any function but in practice the hiddle layer needs to be so large that it is computationally impossible so instead we can add mutiiple hidden layers

Question 6

What is the equation for mean squared error?

Answer 6

E(x) = 1/2 ∑ (y_n - t_n )^2

Question 7

Why do we use a sigmoid function?

Answer 7

For multiple layers we cannot use step functions because it is not differentiable. A sigmoid function aproximates a step function and is differentiable

Question 8

What is the equation for the sigmoid?

Answer 8

f(x) = 1/(1 + e^βx)

β = Changes the steepness of the sigmoid function

Question 9

What is the derivative of the sigmoid function?

Answer 9

σ’ = (βe^-βx) / (1 + e^-βx)²

Question 10

What is the sigmoid activation function?

Answer 10

y(w^Tx) = 1 / (1 + e^{-βw^T x})