Week 9: Rescorla-Wagner and Temporal Difference Learning

Question 1

There are other related model-free algorithms

Answer 1

such as the temporal difference learning

Question 2

Temporal Difference Learning (2)

Answer 2

• It is also a model-free RL algorithm
• It is different from Q-learning

Question 3

Definition of classical conditoning

Answer 3

a learning process that occurs when two stimuli are repeatedly paired; a response that is at first elicited by the second stimulus is eventually elicited by the first stimulus alone.

Question 4

UCS

Answer 4

Unconditioned stimulus

Question 5

UCS is a stimulus that

Answer 5

leads to an automatic response

Question 6

Neutral stimulus is a

Answer 6

stimulus that does not trigger a response on its own

Question 7

Conditioned stimulus is

Answer 7

a stimulus that was once neutral (did not trigger a response) but now leads to a response

Question 8

Unconditioned response (UCR)

Answer 8

is an automatic response tha occurs without thought when an unconditioned stimulsu is present

Question 9

Conditioned response (CR)

Answer 9

Is a learned response created where no response existed before

Question 10

Pavlov’s Dog Experiment (4)

Answer 10

• Before conditioning, Dog was presented with food (UCS) that lead to automatically trigger a salivation response (UCR)
• Before condtioning, Dog heard a bell rang (NS) which lead to no response from dog
• During conditoning, dog presented with food (UCS) and sound of bell (NS) which lead to salivation (UCR)
• After conditioning, the dog salivated (CR) when he heard the bell rang (CS)

Question 11

Diagram of Pavlov’s dog: classical conditioning experiment,

Answer 11

Question 12

Now we view S as

Answer 12

conditoned stimulus (bell after pairing)

Question 13

Do not confuse ‘S’ as

Answer 13

state

Question 14

Now we view R as

Answer 14

reinforcement (i.e., the food)

Question 15

Table of acqusition, extinction, partial reinforcement in classical conditoning Diagram

Answer 15

Question 16

Acqusition in table is where (3)

Answer 16

S (CS) is paired with reward in Phase 1
Nothing in Phase 2
Then we get response to S

Question 17

Extinction in classical condtioning table outcome is where (3)

Answer 17

S (CS) paired with R In Phase 1
Present S (CS) on its own in Phase 2
Then see no response to S

Question 18

Partial Reinforcement in table of classical condtioning (2)

Answer 18

where we ocassionally present S with R
Lead to weak response to S

Question 19

Simple way to model that table of classical condtioning (CC) (of acquisition, extinction and partial reinforcement is) (2)

Answer 19

Stimulus neuron which has an input weight to a reward neuron

If the r neuron is active, we predict reinforcement reward/punishment

Question 20

Use a simple delta-rule model in simple model of CC (of acqusition, extinction and partial reinforcement)

If S stimulus is present… (2)

Answer 20

S=1 (S=0 if not present):

then update weight: w → w + εSδ

Question 21

δ

Answer 21

is delta

Question 22

ε

Answer 22

epilson , a very small number

Question 23

What is δ in simple delta-rule model of simple model of CC

w → w + εSδ

(4)

Answer 23

δ = r - wS,

i.e. the difference between actual reward and predicted (wS) reward.

This is error-driven learning = changing w such delta = 0 in presence of stimulus S

The error between predicted and actual reward is 0 so prediction is perfect and know what to expect

Question 24

Diagram of graph of simple model of CC using simple delta rule model ( explaining acqusition, extinction and partial reinforcement (3)

Answer 24

When the weight grows slowly = acqusition phase

When we do not present rewward the weight decays = extinction.

Partial reinforcement

Question 25

Diagram of 2 stimuli that predict reinforcement model (2)

Answer 25

We have S1,S2 predict reinforcement

If this neuron active (reinforcement neuron) ,then we predict reinforcement reward/punishment

Question 26

V is the

Answer 26

expected reinforcement r reward based on all stimuli

Question 27

Two ways to model that 2 stimuli predict reinforcement? (2)

Answer 27

Do we calculate delta for each S? (stimuli;CS)

Or do we calculate the sum over the Ss

Question 28

Blocking experiments (3) by Kamin 1969

Answer 28

• if a dog is repeatedly exposed to a tone (the first conditioned stimulus, CS1), together with food (the unconditioned stimulus, US), the dog salivates when the tone is presented (conditioned response, CR).
• After several consecutive conditioning trials, this time with the tone (CS1) and a light (CS2) together with the US, the dog does not salivate/weak response to the light (CS2) when tested separately later.
• Stimulus control by CS2 has then been blocked by the earlier pairing of CS1 with the US

Question 29

Overshadowing vs Blocking

Overshadowing experiments (5)

Answer 29

In the example of blocking, the light was pretrained prior to being compounded with the tone, and subjects learned little about the tone-the added element.

Sometimes, even If there is no prior training of an element of a compound CS, subjects will still learn little about one ofthe elements.

This occurs if one element is more “salient” than the other (other things being equal, a subject trained with a more salient CS will learn more rapidly than a subject trained with a less salient CS).

Ifa light CS is more salient than a tone CS, the effect of pairing a UCS with the light + tone compound will be to strongly associate the light with the UCS (food), with little associative strength developing between the tone and the UCS (the light overshadows the tone)

There will then be no response to S2 (tone) = salivation

Question 30

CC (classical conditioning) can be modelled with

Answer 30

R-W rule

Question 31

Blocking and overshadowing experiments indicate how to model 2 stimuli predicting reinforcement is (3)

Answer 31

Seond way by taking the difference between reinforcement and expected reinforcement (V) given all stimuli

(Reinforcement = general term could be reward or punishment)

Producing a single error term for all stimuli (this is known as Rescorla-Wagner rule)

Question 32

What is Rescorla-Wagner Model? (2)

Answer 32

formal model of the circumstances under which Pavlovian conditioning occurs.

It attempts to describe the changes in associative strength (V) between a signal (conditioned stimulus, CS) and the subsequent stimulus (unconditioned stimulus, US) as a result of a conditioning trial.

Question 33

Expected reinforcement , V, has its

Answer 33

own formula

Question 34

What about temporal sequence of stimuli such that (4)

Answer 34

• Stimulus 1 (CS) in Phase 1 , leads to R (reinforcement = reward/punishment)
• Introduce Stimulus 2 (CS) (S2) which predicts S1
• Test the response to S2
• This is 2nd order condtioning

Question 35

What is 2nd order conditoning?

Answer 35

Second-order conditioning (SOC) describes a phenomenon whereby a conditioned stimulus (CS) acquires the ability to elicit a conditioned response (CR) without ever being directly paired with an unconditioned stimulus (US)

Question 36

2nd order conditioning example (5)

Answer 36

For instance, second-order conditioning can be demonstrated using the following procedure:

a CS1 (e.g., a light) is paired with a UCS (e.g., food) in phase 1;

then CS2 (e.g., a tone) is paired with CS1 (the light) in phase 2.

Tested response to S2?

This will usually result in a CR relevant to the original UCS (food) being evoked by CS2, even though CS2 has never been directly paired with food (e.g., Rescorla, 1980; Rizley & Rescorla, 1972).

Question 37

The Rescorla-Wagner rule only works for

Answer 37

direct associations of S (CS) with R

Question 38

The R-W rule does not work in temporal sequence of stimuli since (2)

Answer 38

there is no ‘r’ in phase 2 as delta rule depends on r!

As our delta is difference between received and expected reward

Question 39

The problem of R-W does not work with temporal sequence of stimuli relates to temporal credit assignment problem again (2)

Answer 39

As time of presentation of S2 we don’t know if it will lead to reward

We don’t know past actions were pviotal for good outcome

Question 40

In order to know which of the past actions was pviotal for a good outcome we need (temporal credit assignment problem) - (2)

Answer 40

to consider a time a stimulus/state occured that is predictive of future reward

i.e., Linking classical conditoning with RL (reinforcement learning)

Question 41

Model that links RL and CC together is called

Answer 41

temporal difference learning model

Question 42

In temporal difference learning model we would need the

Answer 42

We need the value function V at time t (Vt) to predict the sum of future rewards, not just immediate rewards so we can learn S2 predicts S1 which predicts R (reinforcement = general like food reward/punishment)

Question 43

In temporal difference learning model we would need the V(t) to predict the sum of future reward, not just immediate r(t) so we can learn S2 => S1 => R

i.e., we want (2)

Answer 43

the sum of all reward at times tau greater than now (present) (at time t)

decompose V(t) to current reward and estimate of subsequent reward

Question 44

To sum of all reward at times taou greater than now (at time t) we need to do this by using the (3) - Temporal Difference Learning

Answer 44

delta rule to ensure this happens:

Delta at time t is current reward + future reward - expected reinforcement (what I expect)

So delta becomes the difference between (expected) reward now and the estimate of all future reward

Question 45

Comparing delta rule in temporal difference learning to R-W rule

Answer 45

Delta is based on current reward only as its delta is the difference between current reward (reinforcement) and expected reinforcement given all stimuli

Question 46

Our TDL delta equation looks suspiciously familiar to the

Answer 46

Q-learning update formula = maps well to it

Question 47

In T-D learning more generally we don’t need to be (2)

Answer 47

restricted to 1 time step forward

We can introduce further time steps in future and discount them using discount factor = as further way in future uncertainity increases

Question 48

TD-0 first, 0-order is closet to

Answer 48

Q-learning

Question 49

We can extend TD learning to include more states in future e.g., 2 time steps in i.e., - (2) = similar to Q-learning

Answer 49

we can estimate two steps into future and if reward is to be expected we update our V (current estimate value of expected reinforcement)

Same form of update as we had in Q-learning expect it is not on state-action pair updating

Question 50

Blocking and overshadowing experiments demonstrate that

Answer 50

ot all stimuli present during learning subsequently control behaviour