BIS II - Introduction to ANN

Question 1

ANN Applications:

Answer 1

• Spam detection: e.g. in emails
• Times series prediction, e.g. in the financial domain
• Pattern recognitions, e.g. objects, faces, etc.
• Computer games, e.g. Go

Question 2

Basic concept of ANN

Answer 2

• Computer technology that attempts to build computers
• That will operate like a human brain
• The machines possess simultaneous memory storage and works with ambiguous information
• Perceptron: early neural network structure that uses no hidden layer

Question 3

Basic concepts of Neural Networks – A Neuron

Answer 3

• Neurons: cells (processing elements) of a biological or artificial neural network
• Nucleus: the central processing portion of a neuron
• Dendrite: the part of a biological neuron that provide inputs to the cell
• Axon: an outgoing connection (i.e. terminal) from a biological neuron
• Synapse: the connection (where the weights are) between processing elements in a neural network

Question 4

Information Processing in ANN – Process Steps

Answer 4

Different inputs are weighted: input consists of the output of the sending unit and the weight between the sending and receiving units

Propagation function: propagation (summation) function determines how the net input is computed; usually a linear combination is used:

Transformation function (also: activation/transfer function)

1. Computes the internal stimulation (activity level) if the neuron. Based on this level, the neuron may or may not produce an output (fire)
2. The relationship between the internal activation level and the output can be linear or nonlinear

5 types of activation functions: 
1.	Linear activation function 
2.	Linear threshold function 
3.	Binary threshold function 
4.	Sigmoid activation functions 
•	Logistic 
•	Tangens hyperbolicus 
5.	Normally distributed activation function 

Output:

1. Sometimes a threshold function is used
2. Most software packages do not distinguish between activation level and output function

Connection Weights:

1. Are associated with each link in a neural network model and express the relative strength of the input data
2. Are assessed by neural networks learning algorithms

Question 5

Further Concepts in ANN -Topologies

Answer 5

• Topologies: the type how neurons are organized in a neural network
• Network structure: (three layers)
  1. Input layer
  1. Each input corresponds to a single attribute, e.g. income level, age, etc.
  2. Several types of data can be used
  3. Preprocessing may be needed to convert the data into meaningful inputs
  2. Hidden layers
  1. The middle layer of an ANN, that has three or more layers
  2. Each layer increases the training effort exponentially
  3. Output layer
  1. Contains the solution to a problem
  2. Purpose of the network is to compute the output values

Question 6

Basic Concepts of ANN: NN architectures

Answer 6

• Feedforward with backpropagation
• Associative memory
• Recurrent network
• Kohonen’s self-organizing maps
• Hopfield networks

Question 7

Learning in ANN: Learning Process

Answer 7

• Learning:
  1. Done by comparing computed outputs to desired outputs of historical cases
  2. Is defined as a change of wrights between units
• The process of learning involves three tasks:
  1. Compute temporary outputs
  2. Compare outputs with desired targets
  3. Adjust the weights and repeat the process if the desired output is not achieved

Question 8

Basic Concepts of Learning in ANN

• Incremental Learning
• Batch Training

Answer 8

Types of Learning

1. Supervised learning: method of training ANN in which sample cases are shown to the network as input and the weights are adjusted to minimize the error in its outputs
2. Unsupervised learning: a method of training ANN in which only input stimuli are shown to the network, which is self-organizing
3. Reinforcement learning: in contrast to supervised learning the outputs are not shown to the network but the feedback whether the output was correct or not
4. Direct design methods (hardwired systems): the weights are not adjusted. There is no learning in terms of weights modification

Time of learning:

Incremental learning:

1. Weights are adjusted after each sample case
2. Also called pattern by pattern, online learning.

Batch training:

1. Weights are adjusted after all sample cases have been put into ANN
2. Also called epoch/offline learning

Question 9

Learning rules of ANN

Answer 9

1. Learning algorithm: the training procedure used by ANN.
2. Overview over learning rules in ANN:
3. Delta rule
4. Gradient descent
5. Backpropagation
6. Hebbian rule (what fires together, wires together)
7. Competitive learning

Question 10

Delta Rule & Propositional Calculus

Answer 10

• Special form of steepest gradient descent approach
• Also called Widrow-Hoff rule (1960) or Least Mean Square rule
• Based in comparison of desired and calculated output

Question 11

Learning in ANN: XOR function and linear separability

Answer 11

• A single neuron (perceptron) represents a hyperplane in instance
• Linear separability is shown in the graphic +
• A single perceptron is not sufficient to build an XOR classifier -> multilayer perceptron is needed
• Three operations AND; OR and NOT can be represented using a perceptron -> any expression from propositional calculus can be converted into a multilayer perceptron, e.g. 1XOR2 = (1OR2) AND (1AND2)

Question 12

Learning in ANN: Backpropagation

Answer 12

Delta rule apply for ANN without hidden layers

• For some problems, ANN with hidden layers are needed, e.g. for solving the XOR problem
• Problem: the desired activation levels (outputs) of hidden layers are unknown
• Backpropagation:
  1. The error (similar to delta rule) is propagated back
  2. But the calculation of the weight changes for hidden layers is also possible

Steps in Backpropagation:

1. Initialize weights with random values and set other parameters
2. Read in the input vector and the desired output
3. Compute the actual output via the calculations, working forward through the layers (Forward-pass)
4. Compute the error
5. Change the weights by working backward from the output layers through the hidden layers (Backward-Pass)

Question 13

Learning in ANN: Gradient Descent

- Limitations & Workaround

Answer 13

• Find the combination of all weights w, so that the sum of the squared error F is minimized
• Problem: high computational complexity in high dimensional spaces
• Solution: steepest gradient descent method
• The gradient is a n-1-dimensional vector of partial derivatives
• The negative gradient gives the direction where to move in the next iteration
• Premise for usage: differentiable propagation, activation, and output functions
• Limitations of gradient descent:

1) Local minimum
2) Plateaus
3) Skipping of minima
4) Direct oscillation
5) Indirect oscillation
6) Saddle points

Question 14

Workaround for Limitations of Gradient Descent

Answer 14

• Workaround for limitations:
  1. Change initial weights
  2. Change the starting point of the gradient descent approach
  3. Change the type of initialization
  4. Change learning parameters
  1. Increase learning rate
  2. Decrease learning rate
  3. Vary learning rates
  5. Define different learning rates for different layers
  6. Insert momentum parameter
  7. Apply simulated annealing

Question 15

Development Process of an ANN

Answer 15

1. Data Collection and preparation: data used for training and testing must include all the attributes that are useful for solving the problem
2. Separate into training and test set: split available dataset into two subsets, one used for learning, the other for testing the learned model
3. Selection of network structure: selection of a topology, i.e. the way in which neurons are organized in a neural network. Determination of:
   o Input nodes & output nodes
   o # of hidden layers & # of hidden nodes
4. Learning algorithm selection: identify a set of connection weights that best cover the training data and have the best predictive accuracy
5. Set parameters and values, initialize weights: set parameters such as weights, learning rate, etc.
6. Transform data into network outputs: define transfer function, e.g. sigmoid
7. Network training: an iterative process that starts from a random set of weights and gradually enhances the fitness of the network model and the known data set. The iteration continues until the error sum is converged to below a preset acceptable level-
8. Testing:
   o Comparing test results to actual results
   o Text plan should include routine cases and potentially problematic situations
   o If the testing reveals large deviations, the training set must be reexamined, and the training process may have to be repeated
9. Implementation of an ANN:
   o Often requires interfaces with other computer-based information systems and user training
   o Ongoing monitoring and feedback to the developers are recommended for system improvements and to ensure that the system is accepted and used properly

Question 16

Advantages & Limitations of ANN

Answer 16

Advantages

• Able to deal with (identify/model) highly nonlinear relationships
• Not prone to restricting normality and/or independence assumptions
• Can handle variety of problem types
• Usually provides better results (prediction and/or clustering) compared to its statistical counterparts
• Handles both numerical and categorical variables (transformation needed!)

Limitations

• Are deemed to be black-box solutions, lacking expandability
• Hard to find optimal values for large number of network parameters
• Optimal design is still an art: requires expertise and extensive experimentation
• It is hard to handle large # of variables (esp. the rich nominal attributes)
• Training may take a long time for large datasets; which may require case sampling