Lecture 16: Decision Making and Neuroeconomics Flashcards
Decision Making:
simple choice
Decision making is a process that results in the
commitment to a choice among options
What is NEUROECONOMICS?
Neuroeconomics studies decision making by using a
combination of tools from behavioral economics, psychology and neuroscience to avoid the shortcomings that arise from a single-perspective approach.
Expected Value
Probability of Reinforcement
x
Magnitude of Reinforcement
Temporal discounting
compare and contrast neuroeconomics and behavioral economics
- Like behavioral economics, the notion of subjective value is important – the value assigned to an object (or choice option) based on choice behavior as opposed to an inherent property of the object.
– Examples, loss aversion, temporal discounting – expected value vs. subjective value - Unlike behavioral economics, neuroeconomics attempts to determine the complex psychological and brain processes/mechanisms that underlie decisions
Factors in the Decision Process
–Internal/external states
or context
– Long-term goals
– Factors contributing to
valuation (e.g., emotion)
– Learning from past
actions
– Memory
Reward:
a key component of valuation
- A pleasurable event that follows a specific behavior.
- Can be primary (eg, food, sex) or secondary (eg, money).
- The brain uses rewards to learn, choose and prepare/execute goals
The function of reward:
–1) Elicit approach behavior (either through innate mechanisms or learning).
–2) Increase the frequency and intensity of a behavior that leads to a reward (learning).
–3) Induce subjective feelings of pleasure (hedonia) and positive emotional states
Dopamine and Reward
Many rewards lead to release of dopamine
(DA).
Dopamine is released during basic drives (i.e.,
hunger, sex).
DA is released in the rat right before and
during copulation, but not afterwards.
DA is released in the human when
presented with food stimulation in a food-
deprived state.
DA is released when participants are
playing video games for mone
The addicted brain
Dopamine and Reward in the Brain
Value representation in the human brain
Meta analysis of 206 fMRI studies examining neural correlates of subjective value
Two regions show a linear increase with subjective value:
Striatum and Orbitofrontal Cortex
Role of the Striatum
The striatum plays a key role in learning
from reward value
The striatum plays a key role in coding
prediction errors allowing learning from
reward feedback to influence future choices
Reward Prediction Error
Dopamine neurons are active when drop of
liquid is delivered outside any behavioral task.
Earliest predictor of reward signals dopamine
response instead of fully predicted reward
Dopamine neurons will be depressed at the
time of the predicted reward if it fails to
occur.
DA provides an error prediction signal to aid
in goal-directed behavior.
prediction error hypothesis of dopamine
The idea: Dopamine encodes a reward prediction error
prediction error hypothesis of dopamine
p equals probability stimulus predicts delivery of reward.
No response to reward that is fully predicted (p=1),
response shifts to stimulus
Do reward prediction errors drive learning in
humans?
Reward learning in humans
Patients
w/ Parkinson’s disease
Loss of dopamine inputs to the striatum
Patients w/ Parkinson’s disease
FMRI
BOLD signal in striatum correlates parametrically, trial-by-trial with prediction
error (O’Doherty et al. 2003)
Learning reward value
Prediction Error:
A general learning signal across
types of decisions
Example:
Social Reputation Learning
or
learning to trust in a behavioral economics games
The Trust Game
Hyperscanning:
two brains interacting &
learning to trust (or not)
Striatum Response to Reward Feedback During the
Trust Game
Responses to monetary reward
in the striatum diminish and
shift to the presentation of
the trustworthy partner as
reputations are acquired
(and feedback is less
informative) during the
Trust game, i.e., prediction error coding
Orbitofrontal Cortex
- Clearly important in driving optimal
decisions, but what is its role? - Evidence from Patients
- Evidence from fMRI
Phineas Gage
Phineas Gage, (born July 1823, New Hampshire, U.S.—died May 1860, California), American railroad foreman known for having survived a traumatic brain injury caused by an iron rod that shot through his skull and obliterated the greater part of the left frontal lobe of his brain.
Gage’s case is considered to be one of the first examples of scientific evidence indicating that damage to the frontal lobes may alter personality, emotions and social interaction. Prior to this case, the frontal lobes were considered silent structures, without function and unrelated to human behavior.
Neural Systems of Decisions
Orbitofrontal cortex: patient studies
- Base of frontal lobe resting on the orbit of the eyes
Often broken down into 2 regions:
– Ventromedial prefrontal cortex (in red) - This region was damaged on Phineas Gage
– Lateral-orbital prefrontal cortex (in green)
- Precise behavioral function is difficult to define
– “irreverent, indulging at times in grossest
profanity … manifesting but little deference for
his fellows, impatient of restraint or advice when
it conflicts with his desires … devising many plans
of future operation, which are no sooner
arranged than they are abandoned in turn for
others.”
– “Gage is no longer Gage” - Types of behaviors related to human orbitofrontal
cortex may not be clearly analogous in other species - Patients with damage to this region seem to have
difficulties with social and emotional “decision making”
Orbitofrontal Cortex Susceptible to Damage
With Traumatic Brain Injury (TBI)
-fill an empty skull with gelatin
-violently rotate the skull
around (simulating a crash or
some type of head trauma)
-most of the “gelatin” brain remains smooth in appearance, except the orbitofrontal region.
-caused by jagged ridges around the eye sockets, provide
protective support for the eye orbits…
Neural Systems of Decisions:
Orbitofrontal cortex
patient studies- Patient E.V.R.
-orbitofrontal damage
-could generate solutions to problems (e.g., two roommates trying to decide what show to watch)
-could not distinguish which solutions were most likely to be effective
-Damasio hypothesized that emotion was necessary to optimize decision making (“somatic marker hypothesis”)
-showed some evidence of flat affect
James-Lange theory of emotion
emotion is your
interpretation of your body’s physiological response
Damasio’s Somatic Marker Hypothesis (p. 400, Ward)
emotional information (physiological arousal) guides decision making
-somatic marker = emotional reaction in your body (physiological arousal)
-hypothesis: orbitofrontal region associates situations and somatic changes
-current options evaluated based on likelihood to be rewarding
Initial Support for the Somatic Marker Hypothesis
-Healthy people, or those with
damage to brain regions
outside the orbitofrontal
cortex learn to draw from the
correct pile (pile B; drawing
from pile A will result in net
loss)
-Are they learning by doing
the math, tracking the
probabilities/expected values?
-Or does one pile “feel better”
after a while, and so people
“go with their gut”?
-Healthy people, or those with
damage to brain regions
outside the orbitofrontal cortex
learn to draw from the correct
pile (pile B; drawing from pile
A will result in net loss)
-Patients with orbitofrontal damage do not figure out which pile to draw from. They also do not show increased GSR for risky draws.
-they also show an increased
galvanic skin response when
anticipating a risky draw (from
pile A)
-Suggests that failure to generate somatic marker impairs ability to learn
the task….(what more would be required to convince you?)
Evidence Against Somatic Marker Hypothesis
Patients with spinal cord
injuries (no physiological
feedback; i.e., no somatic
markers) have normal
emotional experience and can
learn the Iowa Gambling Task
just fine…
Difficulty for frontal patients could be
difficulty with updating value of the
decks (changing rules). In fact, if the
deck starts stacked so large losses are
evident early, they learn to avoid “pile
A”.
Neural Systems of Decisions:
Orbitofrontal cortex:
patient studies
Relative Value of the Choice in context:
– One criterion for selection of an action is to weigh cues about appropriate response
in a context. This function may be related to the orbitofrontal cortex (OFC)
– Patients with damage to the OFC show a range of deficits in social decision making
* Utilization behavior: patients will over rely on external, perceptual cues to
choose a behavior even if it is not appropriate to the social context
* Change in personality, irresponsibility, lack of concern for future consequences
or social norms, diminished social awareness and empathy
– Monkeys with OFC damage will show a decrease in social status
– OFC in humans has been linked to antisocial personality disorder and aggression
* Murderers have lower resting baseline metabolism in OFC than non-murderers
– Most patients with OFC damage are not overly aggressive, but poor social decision
making leads to problems in family, job, future planning
Orbitofrontal cortex
Relative pleasantness
orbitofrontal cortex activity increases
as pleasantness of temperature, taste,
odors, physical attractiveness increases
Also, chocolate, but OFC activity to chocolate decreases
after eating chocolate (and participant is sated)
Orbitofrontal cortex
Imaging studies
summary
- BOLD response in the OFC correlates with
perceived pleasantness - BOLD response in OFC correlates with subjective
value, not prediction error - Combined with patients studies, is consistent with a
role for OFC in encoding relative value of choice
– Value considering the social/emotional context
– Value in relation to previous value for the same choice
(necessary to update value)
– Value relative to internal states, preferences (perceived
pleasantness), and long term goals
– All key components of subjective value
Classical economist’s approach to decision-making
People are
rational and should make the choice with the greatest expected
outcome
* However… people are not purely rational
Psychologist’s approach to decision-making
People don’t behave
rationally because there are psychological factors at play– they
use certain heuristics or have biases when processing decisions
* Examples: People are risk averse because they treat gains and losses
differently (i.e., “losses loom larger than gains”)
Neuroeconomics
Combines classical economic theory,
psychological theory, and neuroscience
Value representation in the human brain
Meta analysis of 206 fMRI studies examining neural correlates of
subjective value
* Two regions show a linear increase with subjective value:
* Striatum and Orbitofrontal Cortex
Getting oriented to the striatum
Sriatum is the largest structure of the basal ganglia
* Dorsal striatum
* Caudate nucleus
* Putamen
* Ventral striatum
* Nucleus accumbens (NAcc)
* Olfactory tubercle (OT)
Striatal “Reward Prediction Neurons”
- What do dopamine neurons in the striatum respond to?
- Dopamine Response = Reward Occurred – Reward Predicted.
Striatal “Reward Prediction Neurons”
- What do Striatal neurons respond to?
- Experiment: Monkeys is either given an unexpected reward or
trained to associate a light with a reward a few seconds later - Result 1: When a monkey receives a reward without an expectation of
reward, the neuron fires
Result 2: A monkey is trained to associate a flash of light with a reward.
When the predictor (light) is turned on, the dopamine neuron fires.
The dopamine neuron does not respond to the reward. Dopamine neurons show little response when the reward is predicted and received
- Result 3: When the predictor (light) is turned on, the dopamine neuron
fires. When there is no reward, the dopamine neuron drops below
baseline firing.
Dopamine neurons show decreased activity when the reward is expected but not received (i.e.,
“negative prediction error”)
“negative prediction error”
Dopamine neurons show decreased activity when the reward is expected but not received
- Prediction error
Discrepancy between the expectation or prediction of
reward and receiving the actual reward
* In response to a positive predictive signals, they fire.
* In response to a rewarding outcome, they :
* Increase activity to “positive prediction error” when a reward is received but not predicted
* Show little response when the reward is predicted and received
* Show decreased activity when the reward is expected but not received (i.e., “negative prediction
error”)
* Prediction error signals in the striatum may be useful for learning about
the rewarding stimuli in the world
A study of prediction error in humans
- Experiment setup: participants don’t know how rewarding a specific
shape is. Each time, a shape is followed by nothing or a reward. So, they
learn over time how rewarding something is
Which brain regions vary with reward prediction error
the striatum
- Other evidence for the involvement of dopamine neurons (presumably in the striatum)
in prediction error learning: - Parkinson’s patients are slower at learning from error feedback
- Parkinson’s patients return to normal function with the L-Dopa
- The striatum plays a key role in coding prediction error,
allowing learning from feedback to influence future choices
(across at least the domains of value and social reputation
learning).
Orbitofrontal Cortex and Decision Making
Orbitofrontal cortex (OFC) is an important structure for
emotional decision making
Potential idea for the role of OFC in decision making:
* Somatic Marker Hypothesis (Damasio)
Case Study: Patient E.V.R. (Elliot)
- Damage to OFC
- Displayed dampened, dull emotional responses to situations
- Can generate solutions to social problems and consider the
consequences - … but cannot decide which solutions are most likely to be
effective or which ones to choose - Damasio’s hypothesis: Maybe emotion is necessary to
optimize decision making?
Damasio’s Somatic Marker Hypothesis
Key point: Our emotional reactions (i.e., physiological
arousal) to a situation are “somatic markers” that can guide
our decisions (e.g., “go with your gut”)
- OFC helps ASSOCIATE situations and somatic (bodily) changes
- OFC evaluates possible behavioral responses and their likelihood for reward, based on previous situations that elicited similar patterns of somatic change
- Make decision based on experience that yielded best emotional
response - Support from Iowa Gambling Task
- Support from Iowa Gambling Task
- Draw either pile A (riskier pile, both wins and losses are bigger) or pile
B (safer pile, smaller wins and losses)à pile A results in net loss
Healthy subjects eventually learn to draw from pile B
* Healthy subjects also show increased skin conductance response (SCR) when drawing from pile A
In contrast, OFC patients don’t show increased skin conductance response (SCR) when drawing from pile A, and don’t learn to avoid pile A
OFC patients continue to choose pile A (riskier pile), and do not show SCR response
from pile A
* Suggests that failure to generate somatic marker impairs ability to learn from
the task
- Evidence against somatic marker hypothesis
- Patients with spinal cord injuries don’t have physical feedback,
and thus have no somatic markers
Patients with spinal cord injuries don’t have physical feedback,
and thus have no somatic markers
- However… they learn the Iowa Gambling Task just fine à don’t
need a physiological response to learn
Maybe OFC patients have difficulty with cognitive control
aspects of the task (e.g., changing rules)
- Maybe they just have trouble overriding what they initially
learned, which requires cognitive control - If the deck is stacked such that losses become evident early on, OFC patients can still learn to avoid pile A
Updating Reward Value/Weighing Value in Context
- Key point: OFC may track the relative value of choice
- Value in relation to social/emotional context
- Value in relation to previous value for same choice
(updating value) - Value relative to internal states, preferences
(perceived pleasantness) and long-term goals
Evidence: Greater activity in OFC as pleasantness
increases
OFC and Pleasantness
various temperatures of water on the hands
OFC activity increases as pleasantness of temperature increases
rich and delicious vs boiled vegetable water
OFC activity increases as pleasantness of taste increases
OFC “Reward Neurons”
Thus, OFC may code value relative to the internal/external
context, or code relative value
Summary
* Striatum = prediction error
* Evidence from monkeys
* Evidence from humans
* OFC = relative value of a choice
* Somatic Marker hypothesis
* Updated view
