Reward and Motivation Flashcards
State-dependency of reward
- The value of rewards depend on the state of the receiver. It is dependent on subjective utility.
- E.g. a bottle of water is much more rewarding if you’re stuck in the desert.
Reward
Punishment
Reward
- Stimulus that elicits approach behavior
Punishment
- A stimulus that elicits avoidance behavior
Primary and secondary reinforcers
Primary reinforcers elicit approach behavior because of its implicit, unconditioned value (e.g. food)
Secondary reinforcers are rewards that are conditioned, but they don’t have intrinsic value. E.g. money is a reward that is conditioned, you can’t really use it for anything unless you spend it.
Negative reinforcement
Reinforcement that is due to the removal of a punishment.
Reward pathways
- Nigrostratal pathway
- Mesolimbic pathway
- Mesocortical pathway
These pathways are characterised by having cells that synthesize dopamine. Thereby, most of the dopamine in the brain stems from one of the two.
The nigrostriatal pathway has dopamine synthesising cells in the substantia nigra.
The Mesolimbic and mesocortical pathways have dopamine synthesising neurons in the ventral tegmental area.
See dedicated cards for further explanations.
Nigrostriatal pathway
The nigrostriatal pathway is a bilateral dopaminergic pathway in the brain that connects the substantia nigra pars compacta (SNc) in the midbrain with the dorsal striatum (i.e., the caudate nucleus and putamen) in the forebrain. It is one of the major dopamine pathways in the brain, and is critical in the production of movement as part of a system called the basal ganglia motor loop. Death of neurons in this pathway can lead to Parkinson’s disease.
Using single cell recordings from the substantia nigra of monkeys, Romo and Schultz (1990) showed elevated activation when presented with food reward, and no activation when presented with no reward.
Mesocortical pathway
Like the mesolimbic pathway, the mesocortical pathway originates in the ventral tegmental area (VTA). The mesocortical pathway connects the VTA and the prefrontal cortex.
The mesocortical pathway is essential to the normal cognitive function of the dorsolateral prefrontal cortex (part of the frontal lobe), and is thought to be involved in cognitive control, motivation, and emotional response.
Dysfunction of the pathway is hypothesised to be involved in psychosis and schizophrenia.
Mesolimbic pathway
The mesolimbic pathway originates in the ventral tegmental area (VTA). The VTA has been called the reward center in the brain, but prof. John is skeptical (as always).
The pathway connects the VTA in the midbrain to the ventral striatum of the basal ganglia in the forebrain. The ventral striatum includes the nucleus accumbens and the olfactory tubercle.
The release of dopamine from the mesolimbic pathway into the nucleus accumbens regulates incentive salience (e.g. motivation and desire for rewarding stimuli) and facilitates reinforcement and reward-related motor function learning.
The dysregulation of the mesolimbic pathway and its output neurons in the nucleus accumbens plays a significant role in the development and maintenance of an addiction.
Nucleus accumbens
- Location?
Located at the intersection of caudate nucleus and putamen in basal ganglia
What is the role of the orbitofrontal cortex in reward processing?
- Multiple modalities
- Lesions?
Multiple Modalities
- OFC is a zone of convergence from multiple modalities
- The orbitofrontal cortex (OFC) is a zone of multimodal processing. Studies have shown that pleasurable stimuli from visual, auditory, gustatory or olfactory modalities will increase activity in the OFC, compared to unpleasant stimuli.
Lesions
- Monkeys with lesions in the OFC will select a boring capsule more often than a delicious banana, which shows that they aren’t able to evaluate the rewarding value of stimuli.
- Focus on frequencies: Monkeys with orbitofrontal lesions choose nonfood items much more frequently (parentheses) than sham (control) surgery monkeys.
- OFC lesioned animals fail to adapt choice behavior when a previously rewarded item stops being rewarded
Medial vs lateral OFC
Medial OFC show increased activation in response to positive rewards
Lateral OFC show increased activation in response to punishment
Classical conditioning
Unconditioned stimulus = e.g. food
unconditioned response = drooling
Without conditioning, dogs will drool when they see food.
Neutral stimulus = e.g. a sound
During conditioning, the neutral stimulus is paired with the unconditioned stimuli.
When conditioned, the sound will now be a conditioned stimulus activating a conditioned response. Thus, the dogs will drool based on the sound, because it is conditioned to do so.
Where does reward anticipation show activation in the brain?
Midbrain and striatum
How is reward expectancy evaluated in the OFC?
Like in V1 or M1, it seems like the reward expectancy is encoded by many different neurons which have different tunings to different probabilities of rewards. These tuned activations all contribute to the reward expectancy.
How does the response of dopamine neurons change from unpredicted reward to predicted reward?
And what happens if the reward is predicted but doesn’t show up?
If a reward occurs without any cues, dopamine neurons in the VTA will fire.
If a reward is preceded by a conditioned stimulus in a way where the stimulus allows the brain to predict the reward (E.g. sound precedes food), the dopamine neurons will fire when the conditioned stimulus occurs, but not when the reward is presented.
If the conditioned stimulus is shown without reward, dopamine neurons will fire when the conditioned stimulus is shown, and there will be a decrease in firing when no reward is given.
I.e. the cells aren’t coding the pleasure of an experience, rather they are coding the reward prediction error.
Delay of gratification
E.g. in the marshmallow test, capability of delaying gratification was predictive for a later success in life.
However, the idea of temporal discounting should be taken into account, as the value of a reward decreases with increasing delay time.
Temporal discounting
The value of a reward decreases as delay time increases. E.g. one marshmallow now is as good as two marshmallows in 15 mins.
What is the ‘reward curve’?
The reward curve is a plot showing how different porportions of reward at different delay times create different preferences for ‘sooner-smaller rewards’ and ‘larger-later rewards’.
Can be used to find the point of subjective equality, where 50% chooses sooner-smaller and 50% chooses larger-later, because the temporal discounting equalizes the value of different sized rewards.
What does the discounted value of reward show?
How much the temporal discounting affects the value of a reward.
Representation of Value
- OFC plays a key role in the representation of value.
- More lateral regions of the PFC are important for some form of modulatory control on these representations or the actions associated with them.
More Than One Type of Decision System?
- Marginal Value Theorem
Marginal Value Theorem
- basic principle in their foraging behavior
- The animal exploits a foraging patch until its intake rate falls below the average intake rate for the overall environment. At that point, the animal becomes exploratory
- the cellular activity in ACC was highly predictive of the amount of time the animal would continue to “forage” by choosing the rewarding stimulus. Most interesting, the cells showed the property of a threshold: When the firing rate was greater than 20 spikes per second, the animal left the patch (monkeys, see pic!)
- the BOLD response in ACC correlates positively with search value (explore) and negatively with the encounter value (exploit) regardless of which choice participants made (humans)
Dopamine Activity and RewardProcessing
- what are the main loci of dopaminergic neurons?
- where do they project to? (pathways)
- Substantia nigra pars compacta (SNc)
- project to the dorsal striatum, the major input nucleus of the basal ganglia
- Ventral Tegmental Area
- Mesolimbic Pathway
- travels to structures important to emotional processing, including the nucleus accumbens (ventral striatum) of the basal ganglia, the amygdala, the hippocampus, and the anterior cingulate cortex
- Mesocortical Pathway
- to the neocortex, particularly to the medial portions of the frontal lobe
- Mesolimbic Pathway
Dopamine Activity and RewardProcessing
Dopamine and Prediction Error
- What is the prediction error (PE)?
- Describe its mechanism!
What is the prediction error (PE)?
- a signal that represents the difference between the obtained reward and the expected reward
Describe its mechanism!
- Positive prediction error (PPE)
- the obtained reward is greater than the expected reward
- Negative prediction error (NPE)
- the obtained reward is less than the expected reward
How can it explain changes in dopaminergic responses to reward?
- if reward unexpected (PPE) -> Dopamine goes up
- after some time, no reaction to US but to CS because reward is now associated with CS
- if reward expected but not obtained (NPE) -> no dopamine
- response decreasing if no reward for some time
What is extincion?
- a response previously associated with a stimulus is no longer produced
