L8 - Neural Networks

Question 1

What is a Qualitative definition of Neural Networks?

Answer 1

• they are a system of equations that aims at reproducing the way in which natural biological systems learn
• Its main constituents are layers, neurons, weights, synapsis and activation functions
• neurons are the fundamental stones of the network
• layers are collections of neurons
• synapsis connect neurons in different layers
  
  • there is no theory behind why we should link each neuron to the next layer of neurons

Question 2

What are the different types of Network structures?

Answer 2

The ways in which neurons are connected determines the type of network

• In feed-forward networks, all neurons in one layer are connected to the neurons in the next layer
• We also have recurrent feedback neural networks, with Kohonen self orienting maps
  
  • Neurons can connect to previous layers or the first layer
• Autoencoders
• Deep Learning with combination of alternative network types
  
  • —The larger the network, the larger the number of weights to be optimized and, therefore, the larger the dataset you need for convergence (at least based on my personal experience…)Deep Learning: Very large networks, with complex structures, applied to the analysis of massive amounts of data

Question 3

What is the mathematical intuition between layers of a network?

Answer 3

• Neuron * synapse for everyone connection going into the next neuron

Question 4

Mathematical intuition of the final layer into the output layer?

Answer 4

• Subbing in for each h we can get b(hat) in terms of a and the original weights of the synapse
  
  • Thus b(hat)=g(A;w)
    
    • A (the data) is know but w is to be determined
  • a is the activation function

Question 5

Examples of the activation function?

Answer 5

determines by how much a neuron is activated

Question 6

What is an important result from Neural Networks?

Answer 6

—Hornik et al (1989) in the journal Neural Networks proves that “standard multilayer feedforward networks are capable of approximating any measurable function to any desired degree of accuracy”.

• The neural network function g(hat) can give a good approximation of the ‘g’ relationship in nature

Question 7

How can we get a good approximation from a neural network?

Answer 7

• Turn it into a complex optimisation problem
• —Several optimization algorithms have been developed
  • Gradient descent
  • Levenberg-Marquadt
  • And their variants
• General idea: optimization algorithms proceed via iterations, starting from an initial guess. These iterations are called “epochs” in Matlab
• The weights are w –> the connections –> coefficients in the linear program
• need to minimise a loss dependent on w
• min_w E(L(B,g(A,w))