Basic Of Learning

Question 1

Perceptron

Answer 1

Perceptron, i.e. a FFNN composed of MCP neurons with step or signum activation function
and threshold T

Question 2

What is the correspondent of biological stimulus for a perceptrone

Answer 2

The input pattern

Question 3

For what task a perceptron can be used?

Answer 3

Classification of patterns

Question 4

How is the threshold managed together with the weights?

Answer 4

The threshold is considered as an additional weight of the neuron with a virtual constant input equal to -1

Question 5

Decision boundary: what is and orientation

Answer 5

Place of the values for which the action potential is 0. It is orthogonal to the weight vector

Question 6

Rosenblatt perceptron learning rule

Answer 6

incremental procedure, we start from an initial weight vector
1)the weight vector is iteratively updated using online strategy
2)each pattern k in the training set contributes to the weight increment vector by means of the error signal
3) one iteration of the iterative procedure requires the evaluation of all R patterns
FORMULA

Question 7

When does the rosenblatt perceptron learning rule correct the weight vector?

Answer 7

If and only if a misclassification occurs

Question 8

What kind of problems can the perceptron solve?

Answer 8

Only the linearly separable ones: it must exist the hyper-plane that completely separates the patterns in between

Question 9

How to extend basic perceptron?

Answer 9

1. continuous output
2. non linear continuous activation function
3. Smooth transition near to 0

Question 10

How to pass from continuous output to categorical?

Answer 10

Softmax network (or manual thresholding in simple cases)

Question 11

Error function with c1 activation functions

Answer 11

FORMULA

Question 12

Why using square in error function?

Answer 12

It makes the error positive and penalizes large errors more

Question 13

Gradient descent

Answer 13

It is an optimization algorithm that approaches a local minimum of a function by taking steps proportional to the negative of the gradient of the function at the current point

Question 14

what is the learning rate?

Answer 14

It modulates the amplitude of the gradient vector in gradient descent

Question 15

Delta rule update formula

Answer 15

dw_ij=(t_i-u_i)f’(P_i)x_j

Question 16

Derivative of logsig

Answer 16

f(1-f)

Question 17

derivative of tanh

Answer 17

1-f^2

Question 18

What does it mean to “reinforce the learning”?

Answer 18

Feeding the network several times with the same training set of patterns

Question 19

Problem of the local minima, how to solve

Answer 19

If we start nearby a local minimum we might end up there instead than the global minimum. It depend on the starting guess so starting with a range of different initial weights sets increases our chances of finding the global minimum

Question 20

What is an important problem of linear activation functions ( a part from linearity)?

Answer 20

Suitable for continuous output but may lead to parameter unbound

Question 21

How to initialize weights? Why?

Answer 21

Randomly, in a small range about zero because sigmoid function can easily saturate for great values of the weights

Question 22

Type of weight update?

Answer 22

On-line, BATCH, mini-batch

Question 23

on-line updating

Answer 23

each pattern error contributes sequentially to the weight updating.
The search in the weight is more stochastic avoiding local minima

Question 24

batch updating

Answer 24

implies that all the pattern errors are cumulated before updating the other weights.
Small pattern errors can be smoothed out so it is less sensitive

Question 25

mini-batch updating

Answer 25

Use of a subset S of the overall training dataset

Question 26

possible stop criteria

Answer 26

1)maximum number of iterations
2)euclidean norm of the gradient vector less than a predefined threshold
3)error function less than a predefined threshold
4)hybrid criterion

Question 27

Regularization factor

Answer 27

norm(w)^2: keeps weights small as much as possible. scaled by a regularization rate and summed to the error

Question 28

heuristic rules for training data

Answer 28

1)training data should be representative for the target task
2)avoiding many examples of one type at the expense of another
3)if one class of pattern is easy to learn, having large numbers of patterns from that class in the training set will only slow down the over-all learning process
4)rescale input values(zero mean and std normalization)

Question 29

How to prevent under-fitting

Answer 29

the network must have a sufficient number of hidden units. Use convergence threshold

Question 30

How to prevent over-fitting

Answer 30

Avoid too much layers and units
Additional noise superimposed to the training patterns
The training can be stopped before convergence