FCFF neural networks Flashcards
Problems of delta rule
1) Does not converge for non linearly separable problems
2) Do not minimiz the number of mistakes
3) No combination of non-linearity
4)For small P the transform is almost linear
Possible task for multi-layer networks in addition to classification
Signal processing, function modeling, model parameter mapping,
Cost functions for regression
MAE, RMSE, RMSEL
MAE formula
sum(abs(t-u))/n
RMSE formula
sqrt(sum((t-u)^2)/n) NB n inside the square root
RMSEL formula
sqrt(sum((log(t+1)-log(u+1))^2)/n)
Binary cross-entropy
-(tlog(u) +(1-t)log(1-u))
categorical cross-entropy
-sum(t*log(u))
What happens in back propagation
Signal error is propagated from output layers back to hidden layers so that all the synaptic weights can be updated
What happens in forward step
Input patterns enter the network
each neuron processes it with the resulting value steadily running through the network, layer by layer, until a result is generated by the output layer. Actual output is compared to the expected output
Backpropagation formula
FORMULA
Vanishing gradient problem
Early hidden layers learn much more slowly than later hidden layers. Weights in the early hidden layers may undergo erratic updating
Solution for vanishing gradient problem
1) Removing fully connection exploiting local receptive field and convolutional neural processing
2)Using autoencoder units
3)automatic weight control of neurons that are saturating during training by means of dropout
What is a problem of linear activation in multilayer network?
The combination of hyper planes generated by internal neurons divides the input space in closed and partially-closed regions characterized by irregular linear bounds
Capability of approximating functions of ffNN
FFNN with a single hidden layer of sigmoidal units are capable of approximating uniformly any continuous multivariate function, to any degree of accuracy
