WK 8 - Neural Networks 1 (AI, The Brain and Neural Computing) Flashcards
Brain Structure
- Contains around 100 billion neurons
- Neurons communicate through synapses – effectively a configurable chemical junction between neurons
Neurons parts
Dendritic tree: receive signals
Cell body: process signals
Axon: transmit signals
Neuronal Function
A neuron receives electrical activity from other neurons along its dendrites (inputs)
The axon (effectively the output of the neuron) will produce a pulse based on the strength of the incoming pulse
This is then passed to other neurons connected to this one
Synapse
chemical junction which can be modified and therefore is thought to be where learning takes place
The synapse can release more neurotransmitter to enhance the coupling between cells
Artificial Neuron
When two neurons fire together the connection between the neurons is strengthened
The activity of firing is one of the fundamental operations necessary for learning and memory
Rosenblatt’s Perceptron Architecture
The Perceptron consists of a single layer of artificial neurons or “perceptrons.”
Each perceptron takes a set of input features and produces a binary output (0 or 1).
Rosenblatt’s Perceptron Training Data:
The training data for the Perceptron algorithm consists of labeled examples, where each example is
epresented by a set of input features and a corresponding target class label (0 or 1).
Rosenblatt’s Perceptron how it works:
the system was able to learn by means of weighted
connections
Problems with the Rosenblatt’s Perceptron
The Perceptron algorithm can only learn and classify linearly separable data.
Binary Classification : The Perceptron algorithm is designed for binary classification tasks, where it assigns instances to one of two classes.
Perceptron could not correctly solve the
XOR function.
Connectionism
Add a further layer of neurons to the network and create a Multi-Layer Perceptron to resolve the XOR issue of teh single layer perceptron.
Neural computing
Neural Computing is based on artificial neural networks (ANNs) that consist of interconnected nodes (neurons) and learn from data through training algorithms. ANNs are inspired by the structure and functioning of biological neural networks in the brain.
Traditional AI:
Traditional AI encompasses various techniques such as symbolic logic, rule-based systems, expert systems, and search algorithms. It focuses on explicit
epresentation of knowledge and logical reasoning.
Artificial Neural Networks (ANNs) learning :
- Supervised Learning
- Unsupervised Learning
Supervised Learning
machine learning approach where:
- artificial neural networks are trained using labeled input-output
-The network then corrects itself based on that output adjusting its internal parameters (weights and biases) during training to minimize the difference between predicted outputs and actual outputs
Unsupervised Learning
-The network organises itself according to patterns in the data
-No external ‘desired output’ is provided
Perceptron
Consists of a set of weighted connections, the neuron
(incorporating the activation function) and the output axon
Modified Versions of Percepron Learning
The larning can be slowed down with a decimal term between 0 and 1 when the weight is updated.
Widrow-Hoff Learning Rule
– weight updates proportionate to
the error made
Δ = desired output – actual output
w_i (t+1) = w_i (t) + ηΔx_i(t)
Limitations of the Perceptron
- can only solve linearly separable
problems ( cannot do it for XOR)
Multi-Layer Perceptron
Single percepron limitation can be overcome by adding a further layer to the network.
Three layers
* Input
* Hidden
* Output
Activation Functions
- Sigmoid function:
The steepness of the curve is changed by z
The derivative can be easily
computed
Weights
- are variable strength connections between
units - propagate signals from one unit to the next
- main component changed during learning
FeedForward - supervised learning algorithm
type of neural network architecture where information flows in one direction, from the input layer to the output layer, without cycles or loops.
Feed Forward output calculation
‒ Multiply incoming signal by weight
‒ Pass this through sigmoid activation function
‒ Pass on this output to units in the next layer
Backpropagation ( updating rule )
Adapt the weights
Start from the output layer and work backwards
New weight (t+1) = old weight, plus a learning rateerror for pattern p on node joutput signal for p on j
Two Types of Weight Updating
Batch Updating : All patterns are presented, errors are calculated, then the weights are updated
Online Updating : The weights are updated after the presentation of each pattern
Momentum
- Momentum encourages the network to make large changes to weights if the weight changes are currently large
- This allows the network to avoid localminima in the early stages as it can overcome hills
weightupdatefunction + (w_ij (t) − w_ij (t −1))
Symbolic AI
- represent knowledge in the form of symbols, rules, and relationships.
- manipulation and processing of symbolic representations of knowledge and logic.
- Expert System (IF-THEN rules)
- Enabling Reasoning
- Knowledge Programmed (by Humans)
- Serial (fragile)
- Does Not Generalise (outside scope)
- Understandable/Explainable
Connectionism
Implicit Representation (numbers)
- Neural Network (weighted graph)
- Enabling Perception
- Knowledge Learned (from Data)
- Distributed (graceful degradation)
- Generalise (outside scope)
- Black-box
Neural Network Properties
-Able to learn to relate input variables to required output
- Is able to generalise between samples
- Shows graceful degradation
Classification
assigning input data to predefined categories or classes.
Regression
involve predicting a continuous output value based on input variables.
Graceful Degradation
symbolic systems: the removal of one component of the system results in failure
Removal of neuron(s) from a neural network: Reduce performance and probably not result in overall failure
‘Generalisation’ in Symbolic AI
Extreamly difficult, Can operate as expert systems in constrained environments but will quickly fail if taken out of environment
Generalisation in Neural Networks
Neural networks can learn common patterns in data therefore they can learn the distinctions between different classes of output
Classification use
designed to group samples according to some known property.
– Minimum 2 datasets required – training and testing
Data Representation Issues
- Continuous data: Good data type for neural networks
- Integer-type Data
-Discrete Categories: Each value needs to have separate representation in the network
categorical variables representations
- Field-type
- Thermometer-type
-Missing Values
Field-type:
Field-type representation is a method where each category is associated with a separate field or feature.
Each field represents a specific category, and the value of that field indicates the presence or absence of that category.
Thermometer-type:
Thermometer-type representation is a method where each category is associated with a “thermometer” vector or bar.
Thermometer-type vs Field
The difference between the two representations lies in the way categories are encoded.
The thermometer-type representation uses binary vectors to capture the degree of membership or association with each category,
while the field-type representation uses separate fields to indicate the presence or absence of each category.
Missing Values
- Occur frequently in real world data
- Cannot be entered directly into the network
- Requires some value in each row
Overfitting
Overfitting occurs when we train the network on a task for too long
The network learns the noise in the input as well as the
common patterns
The result is poor performance on unseen examples
Early Stopping (to avoid overfitting)
- cross-validation
Have three sets, training, testing and cross-validation
