lecture 3 - reinforcement learning

Question 1

value-based decision making

Answer 1

used because many decisions are about objective stimulus values instead of subjective preferences like before (DDM, SDT)

Question 2

reinforcement learning in AI

Answer 2

how do agents learn to behave in an environment

Question 3

reinforcement learning in psychology

Answer 3

how do humans learn from rewards

Question 4

basic mechanisms of interest for RL in neural models for cognitive processes

Answer 4

1. decision-making
2. learning

Question 5

classical conditioning

Answer 5

• a process where a new stimulus-response connection is formed through association, allowing an agent to associate a previously neutral stimulus with an unconditioned response.
• conditioning happens even without action from an agent

Question 6

key stages of classical conditioning

Answer 6

1. Before Conditioning: Unconditioned stimulus (US) elicits an unconditioned response (UR). Neutral stimulus (NS) produces no response.
2. During Conditioning: Neutral stimulus (NS) is paired with US, leading to UR.
3. After Conditioning: The NS becomes a conditioned stimulus (CS), eliciting a conditioned response (CR).

Question 7

acquisition

Answer 7

the process by which a neutral stimulus gains associative value, leading to a learned response

Question 8

extinction

Answer 8

the learned association can weaken and eventually disappear if the conditioned stimulus is not reinforced by the unconditioned stimulus

Question 9

‘kamin’ blocking

Answer 9

a previously learned association prevents the formation of a new association with a second stimulus

Question 10

rescorla-wagner model

Answer 10

The model aims to predict rewards or punishments by learning from the prediction error (𝛿), which is the difference between the expected reward and the actual reward.

Question 11

rescorla-wagner model: formula

Answer 11

prediction error (δ) = actual reward - predicted reward

Question 12

delta-rule

Answer 12

how much learning occurs on each trial, based on;

1. prediction error
2. learning rate
3. stimulus salience

Question 13

delta-rule: formula

Answer 13

ΔV=αβ(λ−ΣV)

Question 14

name the components: ΔV=αβ(λ−ΣV)

Answer 14

ΔV: Amount of learning on a given trial.
α: Learning rate.
β: Salience of the stimulus.
λ: Asymptote of learning (maximum value).
ΣV: Total amount learned so far (expectation).

(λ−ΣV): prediction error (δ)

Question 15

delta rule: prediction error

Answer 15

• difference between the value of the feedback (λ) and the current expectation (ΣV) based on prior experience
• when the prediction error is large, learning occurs more rapidly

if δ is large, the prediction error is larger, and learning occurs more rapidly on the trial than when e.g., δ = 0, as this would indicate no needed adjustment (perfect prediction)

Question 16

delta rule: value of the learning rate

Answer 16

• determines the steepness of the learning curve

1. higher α leads to faster increases in association strength.
2. lower α results in slower learning over trials.

Question 17

delta rule: λ

Answer 17

• asymptote of learning (value of CS)
• larger λ leads to steeper curve, as the initial prediction error will be larger
• extinction happens when λ = 0 (downward curve)

Question 18

delta rule: What happens as ΣV approaches λ over trials

Answer 18

As the sum of learned values (ΣV) approaches the asymptote (λ), the prediction error decreases, learning slows down, and eventually stops when the asymptote is reached.

Question 19

limitation of rescorla-wagner model

Answer 19

• RW model is designed to predict only the immediate reward based on the current conditioned stimulus (CS)
• doesn’t capture more complicated situations such as higher order conditioning, which involves learning complex associations that extend beyond immediate, single-step predictions
• limits its ability to model scenarios where rewards are delayed or involve a sequence of predictive cues.
• we therefore need to predict all possible future rewards

Question 20

temporal difference learning

Answer 20

• extends RW model to cover all the time steps in the trial with an eligibility trace
• we are now not only predicting the immediate reward but also accounting for future rewards

Question 21

eligibility trace

Answer 21

• bookkeeping of all times in the trial
• projects a value back in time, which means that it can adjust earlier state values based on rewards that occur later in a sequence
• enables the model to learn from delayed rewards

Question 22

How does the eligibility trace improve learning in complex scenarios?

Answer 22

It allows the model to adjust earlier state values based on rewards that occur later, making it well-suited for sequential learning and tasks where actions in one state influence future rewards.

Question 23

How is an eligibility trace versatile?

Answer 23

It depends not only on time but also on the state of the world, allowing it to adapt to different learning contexts and scenarios.

Question 24

How does Temporal Difference (TD) learning extend the Rescorla-Wagner model?

Answer 24

TD learning accounts for rewards occurring at different times in a trial, updating state values incrementally

Question 25

What role does the ventral tegmental area (VTA) play in Temporal Difference (TD) learning?

Answer 25

The VTA contains dopaminergic neurons that project to the striatum, playing a key role in reward-based learning by signaling prediction errors.

Question 26

How do VTA neurons respond when there is no prediction of a reward?

Answer 26

When there is no prediction, the conditioned stimulus (CS) does not elicit a response. If a reward occurs, dopamine neuron activity increases, signaling a positive prediction error.

Question 27

What happens in VTA neurons when a reward is predicted and occurs?

Answer 27

Dopamine activity increases when the CS appears (predicting the reward), but there is no change when the reward itself is received, as it was already expected.

Question 28

What happens when a reward is predicted but does not occur?

Answer 28

Dopamine activity increases when the CS appears but decreases (a dip in activity) when the reward fails to occur, signaling a negative prediction error.

Question 29

How does VTA dopamine activity associate with the conditioned stimulus (CS) rather than the reward itself?

Answer 29

The dopamine response shifts from the reward to the CS over time, as the CS comes to predict the reward. This demonstrates TD learning’s ability to adjust expectations based on timing and prediction.

Question 30

How does TD learning in the brain relate to the prediction error concept?

Answer 30

Dopaminergic neurons in the VTA signal the difference between expected and actual rewards (prediction error), adjusting learning and behavior to align with future expectations.

Question 31

Q-learning

Answer 31

• adds action
• models for both reward learning and value-based
  decision-making in humans
• a type of reinforcement learning algorithm that helps agents learn the value of taking specific actions in given states to maximize rewards over time

Question 32

how does Q-learning relate to temporal difference learning, rescorla wagner, and DDM?

Answer 32

1. Models (temporal) reward learning similar to TD/RW learning
2. Models decision-making similar to DDM

Question 33

what is Q in Q-learning

Answer 33

the value of the stimulus

Question 34

Q-learning algorithm

Answer 34

1. initialize Q-table
2. choose an action a
3. perform action
4. measure reward
5. update Q

• repeat steps 2-5 until training is done
• over time, the Q-value for each action is updated as the agent learns from rewards received after taking actions

Question 35

ΔQ-value

Answer 35

• (Q_chosen) - (Q_unchosen)
• if ΔQ is negative, the unchosen stimulus had the highest Q value.

Question 36

choice function

Answer 36

• determines the probability of selecting a particular option t based on its Q-value.
• i.e., decides which option to pick based on the Q-values calculated by the value function

Question 37

choice function: beta parameter

Answer 37

• temperature parameter, which controls the exploration-exploitation balance
• controls how sensitive the model’s choices are to Q-value differences

Question 38

low beta

Answer 38

• more exploration (random choice)
• smooth sigmoid curve for x=Qvalue, y=p(correct)
• fluctuating Q-values since it learns from a wider range of experiences
• fluctuating RPEs due to random exploratory behavior

Question 39

high beta

Answer 39

• more exploitation (deterministic, highest Q-value)
• sharp sigmoid curve for x=Qvalue, y=p(correct)
• smooth Q-value evolution
• smooth RPE evolution since the model sticks to predictable decisions based on the highest Q-value.

Question 40

choice function: trait & choice

Answer 40

the exploration-exploitation balance can be influenced by both traits (individual differences in decision-making styles) and choices (specific decisions made during a task).

Question 41

value function

Answer 41

updates the “Q-value” for the chosen action based on whether the feedback was positive or negative

1. If you pick an option and get a reward, the Q-value increases based on a learning rate (α_gain)
2. If you pick an option and get no reward, the Q-value adjusts downward using a different learning rate (α_loss)

Question 42

2 learning rates to update Q through the value function

Answer 42

1. a_{gain}: speed of learning from positive feedback
2. a_{loss}: speed of learning from negative feedback

• i.e., these determine the speed of learning/updating Q-values

Question 43

increasing a

Answer 43

Q-values adjust more quickly, leading to larger swings in value estimates

Question 44

decreasing a

Answer 44

results in slower updates, producing smoother, more gradual changes in Q-values

Question 45

increasing a_{gain}

Answer 45

• speeds learning up
• Q-value differences reach asymptote earlier in the trials, showing faster differentiation between choice options

Question 46

decreasing a_{gain}

Answer 46

• leads to slower learning
• Q-value take longer to reach their asymptotes. the Q-values differences line line rises more gradually, indicating slower value differentiation

Question 47

increasing a_{loss}

Answer 47

• makes the model ‘unlearn’ Q-values quickly after a negative outcome
• the asymptotes in the Q-values go down
• the Q-value differences line becomes more jagged, as strong updates from negative feedback create more fluctuations in learning

Question 48

decreasing a_{loss}

Answer 48

leads to negative feedback having less influence, so Q-values decrease more slowly after non-rewards. this makes Q-value differences smoother and less volatile over time.

Question 49

reward prediction error (RPE)

Answer 49

• difference between expected and received feedback
• also drives learning

Question 50

What changes when running the Q-learning simulation multiple times with the same parameter settings?

Answer 50

• sequence of trials and specific trial outcomes (e.g., rewards and the precise pattern of reward prediction errors)
• this randomness reflects the fact that a single parameter setting generates data from a distribution of possible outcomes rather than a fixed dataset.

Question 51

What remains the same when running the Q-learning simulation multiple times with the same parameter settings?

Answer 51

• Q-value differences and evolution of reward prediction errors
• these consistent patterns confirm that the parameter settings shape the underlying probabilities governing learning and decision-making.

Question 52

DRL: reinforcement learning

Answer 52

• an agent interacts with an environment by taking actions and receiving rewards based on the state of the environment.
• This approach works well for simple problems with a limited number of states but becomes infeasible as the environment’s complexity grows.

Question 53

DRL: deep learning

Answer 53

used for categorization problems, where a (supervised) classifier learns to categorize inputs based on labeled examples.

Question 54

DRL: deep reinforcement learning

Answer 54

• combines deep learning and reinforcement learning to handle complex RL problems.
• Instead of using a tabular approach, DRL uses deep neural networks to approximate the action-value function or policy, making it scalable to environments with high-dimensional state spaces.