Neural networks for reinforcement learning

Question 1

What neurological substrate do reinforcement models typically concern

Answer 1

Unit recordings from mesencephalic DA neurons in monkeys:
Can we explain their firing from models of Reinforcement Learning?

Question 2

How do these DA neurons behave before learning?

Answer 2

DA cell responds to reward but not to the predictive CS (sound)

Question 3

How do DA neurons behave following learning?

Answer 3

DA cell does not respond to reward when it is predicted by the CS; Backwards shift of response towards the CS itself!

Question 4

Does this change if the reward comes unexpectantly?

Answer 4

Cells still responsive to reward when it comes unexpectedly

Question 5

What temporal Dynamics are at play here?

Answer 5

There’s a fixed interval between sound and liquid reward, Sound is predictive of reward

Question 6

What type of neural network architecture does this model employ?

Answer 6

(iii) Hybrid neural network: feed-forward and feedback/ recurrent connections
* Important subclass: reinforcement learning

Question 7

Describe the architecture of a simple reinforcement learning network subclass of hybrid neural networks

Answer 7

Inputs (P1-5) provide semi connected inputs to:
Hidden layer (3 nodes) provides semi connected inputs to:
Output layer (a1,a2) provides input to:

The environment which both:
Delivers neutral sensory stimuli and
delivers reinforcement (punishment or reward) to post input layers

Question 8

What could be biological correspondence to these variables?

Answer 8

p = sensory patterns such as sound
a = output, let’s say motor output
reward could be prey that was caught and eaten; punishment could be pain

Question 9

What are some key features of reinforcement learning? (5)

Answer 9

• Instructive signal for learning is one scalar value for whole
  network: the Reinforcement signal
• Scalar value can be 1 bit (‘right or wrong’, 0 or 1) or can be
  graded (‘pretty good…very good’)
• Reinforcement Learning follows operant (instrumental) conditioning, but can also be applied to Pavlovian Conditioning
  Stimulus -> Action -> Reinforcement => Modification of
  network connections
• Reinforcement Learning relies on learning with a critic” (was
  the action good or bad?)
• Scalar feedback: only tells how good/bad the action given the stimulus was

Question 10

What can this learning with a critic be contrasted with? What method does this correspond to?

Answer 10

Reinforcement Learning relies on learning with a critic” (was
the action good or bad?)
Contrasts with: “learning with a teacher”
(what was right or wrong in any trial); backpropagation supervised learning

Question 11

What can this scalar feedback not tell us?

Answer 11

Scalar feedback only tells how good/bad the action given the stimulus was ( just one numerical number; 0,1), not what the optimal output would have been

Question 12

What is meant by the credit assignment problem in reinforcement learning? How can this be subdivided? (2)

Answer 12

• In real (and artificial) life, a reinforcement is usually obtained only after a long sequence of actions (e.g. playing chess – win/lose)
• temporal credit assignment problem: which individual move was particularly good or bad?
• structural credit assignment problem: which individual neuron
  (unit) behaved correctly or erroneously?

Question 13

In regards to the taxonomy of mammalian memory systems, where does reinforcement learning concern?

Answer 13

Non-declarative (implicit):RL resorts under stimulus-response skill learning (procedural learning) and classical conditioning

Question 14

What neural substrates are often assigned to these kinds of learning?

Answer 14

Procedural learning: Striatum
Classical conditioning:
Emotional responses: Amygdala
Skeletal musculature: Cerebellum

Question 15

Give the learning sequences for these types of learning respectively

Answer 15

skill: stimulus => action => outcome
classical cond: stimulus => outcome

Question 16

Classical Reinforcement Learning captures only two elements of complex processes underlying operant conditioning, what are these? Describe their learning sequences in regards

Answer 16

Stimulus-response (operant) learning and Pavlovian association

Both concern the transition of a stimulus to a reinforcer

Pavlovian learning assigns motivational value to stimulus and elicits automated (‘reflexive’) reaction (no instrumental action needed to obtain outcome)

Stimulus-response (operant) learning first concerns the transition of a stimulus to a response and a response to a reinforcer

Question 17

Experimental Psychology produced evidence for additional learning processes within stimulus-response (operant) learning, what are these? (The transitional processes )

Answer 17

• Habits: in real life stimulus-response learning eventually leads to habit formation (=weakly sensitive to reinforcement);

*Action-outcome learning (associating a (associated) response to a reinforcer)

Question 18

What behaviour is associated with action-outcome learning according to Pennartz?

Answer 18

Goal Directedness to determine if response is needed for reinforcer

In pavlovian response is not important

Question 19

How does this learning process contrast with backpropagation?

Answer 19

Critic evaluates whether the action is good or not rather than at the individual level; was the action good or not. Backprop would be learning with a teacher, quite artificial.

Question 20

What is action-outcome learning about? (3)

Answer 20

:: Knowing what you need (most) and how to get it
:: Representing your goal before undertaking action
:: Knowing whether your action is relevant or not

Question 21

Illustrate the importance of action outcome learning with a dilemma

Answer 21

“Castaway’s dilemma”; Why stimulus-response learning is not sufficient

A castaway on an island sees palm trees, what action should he carry out?
One stimulus, multiple options:
1) Search for coconuts?
2) Burn trees to get warm?
3) Build a raft to escape?

Stimulus-response learning does not solve the problem; Stimulus does not tell you what to do. Once goal is identified, you need to know the associated action required to achieve that goal

Question 22

What conclusion ca be derived from this dilemma?

Answer 22

Conclusion: simple (Pavlovian) RL captures only part of complex forms of learning (but it’s a good start)

Question 23

Name a different type of reinforcement learning related to GDP

Answer 23

Temporal Difference (TD) Learning

Question 24

What is TD-RL learning suitable for?

Answer 24

multi-step tasks

Question 25

Give an example illustrating an application TD-RL networks

Answer 25

Example of multi-step task: Tower of London task

• Requires planning of multi-step operations
• Typically depends on prefrontal cortex
• Can be successfully modelled using ‘Temporal Difference’ RL models

Question 26

Describe the use of error in TD-RL networks

Answer 26

Error in reward prediction

Instead of direct feedback by actual reward, it is more efficient to use an Internal feedback signal. In practice, each task step can be associated with a reward prediction; the internal feedback signal is often an error in reward prediction:

𝛿 = (R –V)

with 𝛿: error in reward prediction
R: real, actual reward (at the end of the task trial)
V: predicted reward (=expected reward; expected value, or just: value)

Question 27

Where does this concept of TD-RL networks data back to? What was posited?

Answer 27

Concepts date back to Robert Rescorla & Allan Wagner (1971, 1973)

• The expected reward fluctuates over time: Value function, V(t)

Question 28

Describe the structure of the temporal difference learning network

Answer 28

Visual stimulus enters the sensory network. The subsequent reward prediction is sent to a node alongside the actual reward. The error in reward prediction is sent to the motor network so a response can be carried back AND back into the sensory network where a learning rule preposterror is carried out. The change in reward prediction (V(t)) over time ins then projected back to the node with the actual reward.

Question 29

What applications can TD learning have outside neuro?

Answer 29

Computing Values & Errors, now and in the future; finance, covid numbers

Question 30

How is the value function defined?

Answer 30

V(t) = E[γ^0.r(t) + γ^1.r(t+1) + γ^2.r(t+2) +…]

E[..] Expected value of the sum of all future and current rewards

r(t) actual reward at time t

γt discount factor: makes “early” rewards (at t) more important than rewards that arrive later (t+1); is smaller than 1.0

Question 31

What does the value function allow the agent to do?

Answer 31

Agents may not know exactly what kind of rewards will come, but can estimate the future value of the situation at t: V(t)

Question 32

How is the value function based on a cue and reinforcer?

Answer 32

Value function: based on temporal relationship between sensory cue (CS) and reinforcer (r)

CS => delay: 2 sec. => Reinforcer (classical conditioning)

• The cue directly activates the network, which is trying to generate a good estimate of future rewards: V(t)
• This prediction can only be improved when learning takes place (change synaptic weights)

Question 33

The function V(t) is consistent over time, give an equation that demonstrates this

Answer 33

V(t) = E[r(t) + γV(t+1)]
V(t+1) represents all rewards expected after time t

The equation presents a fully learned situation: you know how much reward to expect after time = t

Question 34

How do you compute the error in the reward prediction?

Answer 34

Bring the V(t) to the right-hand side; in case of perfect learning, zero would remain at the left-hand side

Imperfect learning means there is error in the reward prediction:
𝛿(t) = r(t) + γV(t+1) - V(t)

with:
𝛿(t): error in the reward prediction at time t
r(t): actual reward at time t
g:discount factor (makes t+1 slightly less important than t)
V(t) estimate of all future reward at time t
V(t+1) estimate of all future reward after time t (from t+1 onwards)

This is why the algorithm is called Temporal Difference Learning (t vs. t+1)

Question 35

Give 3 relevant cases if no reward is expected after time t

Answer 35

1) reward present, but not predicted (V(t)=0) => 𝛿(t) > 0
2) reward absent, although predicted => 𝛿(t) < 0
3) learning complete: reward correctly predicted => 𝛿(t) = 0

Question 36

How can you relate these rules to the network structure?

Answer 36

If error is 0, the prediction from sensory network is equal to sensory reward

+ error –> stronger synaptic connection needed –> until there is a match

Question 37

What is still missing from this network compared to the equations?

Answer 37

Time is still a problem, usually stimulus is earlier than reward (could be hours between)

A significant Delta (error) can occur even before an actual reward has been received; this is called a “surrogate prediction error”.

e.g. r(t)=0 but gV(t+1) is large & positive and V(t)=0; learning will occur !

Question 38

What could fulfil this role in the network?

Answer 38

Temporal Difference: role of dopamine? Could the role of “error-coding unit” fulfilled by Dopamine cells? (hypothesis)

Question 39

Relate this concept to an application

Answer 39

An animal has learned that cue X (at t+1; deer) precedes reward by a short delay, but not yet that another cue Y (at t; scent) precedes reward by a slightly longer delay

Question 40

What happens in the network as a result of the surrogate prediction error (d(t) > 0)?

Answer 40

• a value will be assigned to cue Y occurring at time t (input layer; cue Y = node )
• this assignment results in altered synaptic weights of neurons responsive to y (in hidden layer):

synaptic change = presynaptic input y * postsynaptic activity * error

Question 41

What does this mean temporally during learning?

Answer 41

Thus, during learning, attribution of value occurs ‘backwards in time’

(“backwards referral”; because Y
precedes X)

Question 42

What physiological network could this be mimicking?

Answer 42

Mesolimbic dopamine projections (VTA, ventral tegmental area) to striatum and frontal cortex

Ventral striatal back projections then project back to VTA

Question 43

Describe research that investigates this relationship between reinforcement learning and this network

Answer 43

Same as previously with the unit recordings from mesencephalic DA neurons in monkeys:

Before learning:
DA cell responds to reward but not to the predictive CS (sound)

After learning:
DA cell does not respond to reward when it is predicted by the CS
Backwards shift of response towards the CS+ !

Evidence in favour of the TD model

Question 44

Fully trained situation: Do the findings confirm a TD model of DA cells?

Answer 44

Reward at the expected time does not elicit a response in DA cells

*Reward omission at expected time: ~ decrease in firing
=>agrees with “negative error in reward prediction”

*Reward shifts to an unexpected time: increase in firing
=>agrees with “positive error in reward prediction”

Overall: strong evidence in favour of predictive coding-in-time

Question 45

Give some uncertainties about dopamine & reinforcement Learning models in terms of neuroanatomy

Answer 45

learning rule supposes convergence of DA terminals with glutamatergic inputs on dendrites– is this found?
=> perhaps in the striatum (triads are rare)

“Triadic” configuration: Cortical and dopamine afferents inputting on striatal neuron dendrites
synaptic change = presynaptic actpostsyn.acterror

Question 46

Give some uncertainties about dopamine & reinforcement Learning models in terms of neurophysiology (2)

Answer 46

DA neurons can fire in advance of eye saccades, so before the animal may begin to identify a cue (CS)

DA neurons are more broadly tuned than just to reward or reward-predicting stimuli (e.g., novelty; uncertain outcome; movement; pain)

Question 47

Give some uncertainties about dopamine & reinforcement Learning models in terms of neuroanatomy

Answer 47

DA may be involved in associative learning because of a more general role in sensorimotor processes (e.g. Parkinson’s)

Question 48

Give some uncertainties about dopamine & reinforcement Learning models in terms of cellular neurophysiology

Answer 48

does DA affect synaptic plasticity (e.g., LTP) according to the TD learning rule? => still debated

Question 49

Overall what could you conclude based on this evidence?

Answer 49

DA may have a more general sensorimotor function – reacting to unexpected salient events; still uncertain whether it literally mediates TD-RL

Question 50

What is an alternative to dopamine for these proposed circuits?

Answer 50

Pyramidal Cell networks in cortex & amygdala:

*Reinforcement Learning also possible using Glutamatergic signalling (-> pyr. cells)
*Brain networks (using Glu) need to store more than just stimulus value,
e.g. knowledge on outcome identities (”what”) =>
model-based learning is learning an Internal Model
of the causal relationships between specific stimuli, actions, outcomes

Question 51

In this glutamine model, where could reinforcemet take place?

Answer 51

Medial PFC
Orbitofrontal PFC
Amygdala: can have error coding property according to research

Question 52

Describe study structure for coding of reward and reward prediction in this network

Answer 52

Evidence for coding of reward and reward prediction in dorsolateral prefrontal cortex:

Paradigm for monkeys –> lever-press task –> can choose left or right for reward

Spatial delayed response task: instruction cue coupled to objects differing in reward value (raisins, apple and cabbage. The monkey would prefer raisins or apple to cabbage ) => testing the relative reward value of objects

Question 53

Describe study results for coding of reward and reward prediction in this network

Answer 53

There is coupling between instruction cue and specific outcome observed

if 2 different rewards, one more preferred (A>B, ‘high’) –> when theres a choice, high firing activity for high preferred stimulus but lower firing activity to low preferred stimulus

Once B becomes the preferred food (over an even less preferred food C) –> will be reflected in firing activity through increase in FR

Therefore PFC cells can code relative expected reward

Question 54

How compatible are these proposed DA network?

Answer 54

compatible with DA model –> PFC sends projections to VTA and SN

Compatible with:
a) Dopamine model of TDRL
(here, PFC may code V(t))
a) Glutamatergic model of RL