Week 4: recurrent visual processing

Question 1

What are lateral connections?

Answer 1

Inputs to a unit coming from other units in the same layer

Question 2

For what kind of networks have are temporal inputs important?

Answer 2

Linguistic processing

Question 3

What is a recurrent convolutional artificial neural network?

Answer 3

The convolutional filter is not only applied to a spatially-limited group of units in the previous layer but also to the surrounding units in the same layer, and also possibly other layers

Question 4

Why are biological and recurrent convolutional networks time-dependent?

Answer 4

The activity in a layer changes depending on results of surrounding activity in the layer and activity in the next layer. there is dynamic interaction between layers to reach an equilibrium which is time-dependent

Question 5

How is recurrent activity implemented?

Answer 5

Through lateral and feedback connections

Question 6

What are dynamic neural oscillations?

Answer 6

The dynamic interactions between excitatory and inhibitory neural population produce oscillations in neural population activity. at oscillation peaks, excitatory population activity is highest

Question 7

How is recurrent activity implemented in deep networks and how is this different from biological networks?

Answer 7

In a layer with recurrent activity, the activity of each unit at one time point feeds into the same unit at the next time point. In biological networks neurons never synapse directly with themselves

Question 8

What is a BLT artificial network?

Answer 8

Bottom-up, lateral and top-down artificial network - type of recurrent network

Question 9

What happens to the network’s activity in a BLT network?

Answer 9

The network’s activity moves back and forth because the results of each interaction affect the activity of the interacting elements - no fixed state of activity

Question 10

How do BLT networks change the computational time? why?

Answer 10

Much more computationally intensive

• more extensive filters
• multiple time steps modelled

Question 11

What developments have there been in recent convolutional networks?

Answer 11

Recently developed recurrent convolutional networks are investigating lateral and top-down connections

Question 12

What happens to image classification when using BLT networks?

Answer 12

Image classification performance in difficult tasks improves considerably

Question 13

Why are BLT networks time dependent?

Answer 13

The interactions between different layers make the network time-dependent

Question 14

What is the difference between a deep feedforward network and a shallow recurrent network?

Answer 14

Recurrent connections effectively make networks deeper

• the same transformation is repeated by recurrent cycles
• each layer performs multiple layer operations but each with the same set of weights, fewer weights to learn
• recurrent networks match neural architecture and activity more closely than deeper networks.

Question 15

What happens in attractor/hopfield networks?

Answer 15

Frequently seen patterns of activity produce strong connections between the activated group of neurons due to Hebbian learning. this pattern gets built into the connection weights. new incoming patterns causes a sequence of recurrent activity that activates units even when there is no input to activate then directly. neurons usually activated together in the higher layer become active - an incomplete pattern comes in it is completed based on experience

Question 16

What does the attractor/hopfield network require?

Answer 16

Multiple recurrent cycles

Question 17

How do different networks perform with recognising incomplete or noisy patterns?

Answer 17

Feedforward networks perform poorly and increase linearly as more of the object is shown. A recurrent hopfield attractor network approaches human performance

Question 18

How does feedback work in predictive encoding?

Answer 18

• the initial sweep forward extracts image features
• prediction is generated based on previously seen patterns
• the completed prediction is fed back to the lower layer and inhibits activity that fits with this prediction
• the feedback connection carries the prediction
• the feedforward signal becomes only the parts that don’t fit with the prediction, the prediction error
• remaining activity signals prediction error
• Hebbian learning follows this error signal, like back-propagation

Question 19

What are PredNets?

Answer 19

A recent type of recurrent network used to implement predictive coding. they predict future activity based on current activity and previous experience, but no labels

Question 20

How can predictive coding happen in networks?

Answer 20

It can emerge spontaneously in a network where each layer has a full set of within-layer connections and feedforward and feedback convolution, without blocks explicitly designed to generate predictions

Question 21

Why do researchers try to train a network to minimise pre-activation in a predictive network?

Answer 21

Asking a network to minimise the summed activation of its inputs, the network can’t just set everything to 0 because the input image will produce a lot of activation. it needs to actively inhibit the input image. this is effective for reducing energy consumption and predicting the next input

Question 22

Why might the brain use predictive coding?

Answer 22

To reduce energy demands

Question 23

What is the blue brain project?

Answer 23

A project that has recently finished mapping the full neural circuitry of mice

Question 24

Why were current processes such as convolution and rectification created?

Answer 24

They were chosen for computational efficiency rather than accurately simulating the underlying process

Question 25

What has neuroscience revealed as the basic principles of neural responses and their interactions?

Answer 25

• recurrent interactions
• attractor networks
• predictive coding
• Hebbian learning from prediction errors

Question 26

What are current neural networks good for?

Answer 26

Making predictions about external events, some characteristics of biological understanding and intelligence

Question 27

What is the software toolbox Neuron used for?

Answer 27

biophysical simulation of interacting neurons as mapped from mouse brains. uses the neural parameters and connection patterns actually measured in the mouse brain

Question 28

What are the two problems with making a similar map of the human brain?

Answer 28

1. ethically we can’t make genetically engineered human clones to study
2. the computing power needed to run the simulation of even the mouse brain is not available

Question 29

What is the problem with simulating neural circuitry on a computer?

Answer 29

Most computer processors are designed for linear mathematics
-they can perform any operation accurately but they are slower and more energy-hungry than the brain, so they are currently being implemented in simpler neural models

Question 30

How is the computing power problem being overcome?

Answer 30

Running biophysical simulations in real time required neuromorphic processors, designed to run neural simulations. they are developing quickly and may be ready in 10 years