Reward and learning - Dr McCabe

Question 1

How does a PET scan measure

Answer 1

radiation emitted from a radioactive glucose tracer in the blood - active regions in the brain require greater blood supply and therefore release greater quantities of radiation

Question 2

How is the radiation in a PET scan released/detected

Answer 2

radioactive chemical releases positrons (injected or inhaled)
radioactive tracer flurodeoxyglucose (FDG)

Question 3

advantage of PET

Answer 3

measure many aspects of function in the brain

body treat flurodeoxyglucose in similar way to normal glucose

Question 4

disadvantage of PET

Answer 4

poor spatial res but better than EEG/MEG
poor temporal - relies on blood
injection of radioactive
bulky and costly

Question 5

What does an MRI measure?

Answer 5

signal changes in the brain related to different magnetic properties of ocygenated and deoxygenated properties in the blood - relate to neural activity

Question 6

advantages of MRI

Answer 6

good spatial res

non invasive

Question 7

disadvantages of MRI

Answer 7

worse temporal than EEG/MEG (but better than PET)
expensive
participants cant have metal in body, must stay still and experience can be claustrophobic

Question 8

What is learning and why is it necessary

Answer 8

Learning is an innate behavioural response which enables adaptation in novel situations - necessary for survival of the fittest so can quickly respond to stimuli in the environment

Question 9

basic idea of classical conditioning

Answer 9

unconditioned stimulus provokes an unconditioned response (natural ie startled)
pair with a neutral stimulus that does not evoke a response (ie bell tone)
consistant association between NS and response leads to association of the stimulus with the response - react only to ns
ns becomes conditioned stimulus with conditioned response

Question 10

How might classical conditioning go wrong?

watson and rayner

Answer 10

9m infant test for fear response to range of stimuli
feared loud and unexpected bang - pair with white rat
become fearful of white rat and generalise to other stimuli ie beard

Question 11

basic idea of operant conditioning

Answer 11

behaviour leads to reward or punishment which determines if the behaviour is repeated or omitted
relationship between the behaviour and its consequence

Question 12

basic idea of blocking

Answer 12

when add in another stimulus in CC assoc - dont learn a new association as unnecessary, already have assoc and new stimulus doesnt provide any additional information
LEARNING ONLY OCCUR WHEN SOMETHING HAS CHANGED
not because first stimuli preempts attention but because second stimuli fails to dignal a change in rienforcement

Question 13

when does learning occur after a conditioned assoc has been formed

Answer 13

when something has changed i.e. information expected has been omitted
when make mistakes

Question 14

define a prediction error

Answer 14

learning occurs only when what was expected does not occur and therefore made an error in judgement - ie.e the predicted time or occurance or magnitude of a stimulus is different to what expect/is ommited completely
what actually occurs is not what was predicted

Question 15

define the rescorla wagner model

Answer 15

change in predictive value of a stimulus = difference between what actually happens and what you expected to occur (suprise)

Question 16

what is dopamine

Answer 16

dopamine is a neurotransmitter/neuromodulator in the brain

most predominantly targets the stratium for motivation and action

Question 17

what role does the basal ganglia have with reward learning etc

Answer 17

The basal ganglia, a group of interconnected brain areas located deep in the cerebral cortex, have proved to be at work in learning, the formation of good and bad habits
projects from cortex (sensory and motor) to stratium - lots of connections with dopamine
topographic

Question 18

define topographic

Answer 18

sets of systematic axonal connections from
one neural region to another that preserve (or precisely
invert) the spatial relationship between neurons; cells that
are close together on the sending surface project to regions
that are close together on the target surface

Question 19

functions of the basal ganglia

Answer 19

hought to be mainly involved with aspects of motor control i.e. disorders such as parkinsons - reaching and grabbing problems
BUT also
basal ganglia facilitate learning, with the neurotransmitter dopamine important to the process
dopamine released in the basal ganglia system communicates with theprefrontal cortex to allow people to pay attention tasks, ignore distractions, and update relevant task information in working memory during problem-solving tasks

Question 20

describe dopamine responses before conditioning is learned (schultz et al 1997)

Answer 20

burst of dopamine activity to unexpected reward - no pairing with stimulus

Question 21

describe dopamine response during/ after conditioning (schultz et al 1997)

Answer 21

response begins to transfer towards the stimulus/cue presentation instead of at the reward - fires at the prediction of a reward

Question 22

describe dopamine response when ommit reward expected from previous conditioning (schultz et al 1997)

Answer 22

spike at prediction but activity of dopamine depressed below basal firing rate where expect the reward - recognise something has changed

Question 23

how do dopamine responses change according to probabilities between the stimulus and the reward

Answer 23

0% predictive of cue - spike at reward
50% predictive of cue - spike at cue and reward but both smaller activity
100% predictive of cue - spike only at prediction
dopamine response to reward as a weighted sum of past and current rewards (r-v)

Question 24

what did ramania et al 2004 want to investigate?

Answer 24

look at prediction error related activity in human brain during classical conditioning - fmri used to localise activity in the brain of DA neurons in tracking prediction errors when monetary rewards delivered independent of goal related actions
1- failure of expected rewards and 2- occurence of unexpected reward
control condition where event occurred as expected

Question 25

problem with ramania et al 2004

Answer 25

assume that fmri relate to activity of dopamine neurons byt not specific - only shows the areas active during learning

Question 26

what did ramania et al 2004 find in their fmri study

unexpected omission of reward

Answer 26

depressed activity anterior superior frontal gyrus, FC, temporal pole and superior temporal sulcus
brain recognised omission by reducting
error = depressed activity in ant.PFC and OFC for omission but increased for presentation

Question 27

define tonic release of dopamine/neuromodulators

Answer 27

sustained release over relatively long periods of time associated with general dopamine activity

Question 28

define phasic release of dopamine/neuromodulators

Answer 28

bursts of neuronal firning thought to be involved in learning associations between stimuli and consequences

Question 29

how does dopamine activity relate to the contigencies associated with a reward (fiorrilo 2003)

Answer 29

track the probability that a cue is related to a reward - as probability of assoc changes, so does dopamine activitiy - code the discrpency between expected and actual reward
tonic firing maintained in uncertain trials where probability 50%/0.5
frontotemporal circuit not only process the predictive stimuli and reward but actively encodes the assoc between them

Question 30

activity of midbrain neuron characteristics (fiorillo, 2003)

Answer 30

signal the prediction of a future reward(spike at cue), unexpected occurence of a reward (spike at presentation) and its unexpected absence (pause in activity)

Question 31

what does data show about dopamine and regions of the brain associated with reward learning

Answer 31

anterior prefrontal cortex - responds to both types of prediciton error (unexpected and omitted)
medial orbitofronal cortex - activity changes specific to unexpected reward
activity in frontotempotal circuits actively encode the association between stimuli and rewards

Question 32

define temporal difference error

Answer 32

used in rienforcement learning to predict reward over time
(RESCORLA WAGNER DOES NOT ACCOUNT FOR THIS)
timing in trial taken into account to provide a prediction ofr the time between and stimulus and an expected/unexpected reward -

Question 33

what does temporal difference error imply about reward learning

Answer 33

constantly updating our understanding about the association between a stimulus and a reward
i.e. expect after long/short period of time etc

Question 34

what do temporal difference errors predict (o’Doherty et al 2006)

Answer 34

before learn - positive PE (assoc strength) response to reward (UCS)
during learning shift to the CS (cue)
unexpected reward lead to positive PE response at delivery
unexpected omission lead to negative PE response at expected delivery

Question 35

what did O’Doherty et al 2006 want to investigate

Answer 35

see if human ventral stratium, orbitofrontal cortex etc show activity consistent in temporal difference error predictions using fmri during appetitive conditioning of pleasant taste reward

Question 36

what did O’Doherty do to look at TD error

Answer 36

calculated TD error from behaviour between cs presentation and reward time
use info as regressor in fmri
look at correlation between change in TD error and brain activity
show parts of brain that track TD error

Question 37

what did O’Dohery 2006 find

Answer 37

backward shift in time of peak of hemodynamic response in ventral stratium during learning - response transfers from time reward presented towards the time the cue is presented

Question 38

how did Rolls, McCabe and ReDoute 2008 investigate TD error in a probablistic decision task?

Answer 38

OC -decision influence when high risk for large value reward vs low risk for low reward
ask participants to press left for 10p 90% time or right for 30% either 90%, 70% or 60% of time - told to max winnings
Expected value = choice calculated after choice made ( update over time as calc predicition error between EV and magnitude of reward) - EV x RM+TD -use as regressors to track changes in brain according to fmri

Question 39

Results of Rolls et al 2008 reward magntitude TD errors

Answer 39

TD error correlate with nucleas accumbens, IFG and midbrain acitivty
change behaviour related to contingencies -
RM corelate with OFC
TD error correlate with NA, frontal gyrus and midbrain
EV neg correlate with anterior insula - expect low reward and when uncertain about outcome (50%)
mOFC respond to reward and to prediciton - reflect prob how much reward obtained based on risk taken
VS pos correlate with reward obtained in present

Question 40

how can dopamine info be applied in real world

Answer 40

understand how decisions etc go wrong ie schisophrenia

Question 41

what did Kumar et al 2008 do

Answer 41

compared results of adults with MDD on antidepressant medication (citalopram) with controls of non medicated and acutely medicated patients and scanned with fmri during reward learning task - TD error in CC task
depression assoc w/ anhedonia symptoms - assoc w/ reduce DA functioning

Question 42

what did kumar et al 2008 find - med depressed

Answer 42

depressive have sig reduced reward learning signals in
- VS
- r/d ACC
-retroplensl cotrex
-midbrain
- hippocampus
abnormal TD correlate with illness severity
BUT enhances signal in VTA- compensatory reponse for blunted reward sig outside brain stem or due to meds?

Question 43

define positive rienforcement

Answer 43

rewarded for behaviour to encourage repetition

Question 44

define negative rienforcement

Answer 44

remove neg consequence to encourage repetition

Question 45

define punishment

Answer 45

pos or neg to weaken response to a stimulus

Question 46

when might blocking fail

Answer 46

when the introduction of a second stimulus does signal and increase or decrease in rienforcement/punishment than the first sitmulus alone
ie stronger shock

Question 47

rescorla wagner in the explanation of blocking

Answer 47

novel stimulus has no additional predictive value = 0
first stimulus already established full assoc value with CS = 1
neither CS of novel stimulus changed predictive value because cs already at 1- no suprise

Question 48

DA in prediction error

Answer 48

has been recorded in primates to track and display “error signals” - activity of dopamine neurons appears to follow similar idea presented in rescorla wagner

Question 49

milner and olds 1954

DA system in rat behaviour

Answer 49

electrode in hypothallamic areas in brain - pleasure centre
stimulated when approached certain parts of cage
found rats would return to place where recieved previous stimulation

Question 50

spangel and weiss 1999

DA associated brain areas

Answer 50

self stimulation found in ventra tegmenta area (VTA) which projects to closelet associated limbic structures
thought to be involved in DA system for natural rewards and can be disrupted by drugs

Question 51

koob and moal 1997

DA and drugs

Answer 51

DA recepors have a rienforcing influence on drug use
increase DA in brain which leads to pleasure
BUT downregulates - need more drug to get same effect which leads to an increase in drug use

Question 52

areas within the basal ganglia

Answer 52

caudate nucleas
putamen
globus palidus
substantia nigra
subthallamic nuclease

Question 53

overlapping functions within the BG

Answer 53

attention selection
switching
internal generation of movement
rienforcement learning

Question 54

van schouwenburg 2014 BG processes

Answer 54

in behavioural/attentional switching, BG enhances procesing of attended features and supressess unattended by modulating connection sfrom the PFC (top down) to the visual cortex
BG mediates top down connections

Question 55

dopamine deficiencies and links to disorder

Answer 55

deficiency of DA in nigrostriatal area related to parkinsons

and disturbance thought to also be responsible to schizophrenic systems

Question 56

schultz et al 1997 importance of dopamine in prediction

Answer 56

prediction gives animal time to prepate to a future stimulus and react in an appropriate manner
reward value not static - assign values at tiem stimulus is sncounterd and as a function of continued experience
DA in VTA nad SN assoc with reward process

Question 57

florillo et al 2003 prediction of reward

Answer 57

when reward mag and timing constant, error = prob of outcome compared to actual
monkeys conditions in CC paradigm in prob of stimulus following a liquid rewrd - measure licking beh with increased prob
as prob increase, so does licking behaviour as presentation of predictive cue

Question 58

describe pearce-hall theory of attention and learning

Answer 58

atention and learning is proportional to uncertaintl about rienforcers
DA facilitates attention and learning when uncertain about reward - attention necessary for learning
BUT when establish an assoc, no further attention required

Question 59

bayer and glimcher 2004 prediction error

Answer 59

DA respnse = weighted sum of current and past rewards
examined activity in single DA neurons during trial and error
neurons encode difference between current reward and the WEIGHTED AVERAGE of previous rewards
DA firing increase when current > than weighted av of prev BUT the same when current less than weighted av of previous

Question 60

ramania et al 2004 findings fmri

unexpected reward

Answer 60

activate medial orbital gyrus of OFC, PFC and inferior frontal sulcus

Question 61

kumar et al 2008 findings med controls

Answer 61

TD signal blunted in rACC, RC, and hippocampus
BUT not increase signal in VTA
- enhanced VTA in depressed unlikely to be due to meds

Question 62

pizzagalli et al 2009 depression and reward

Answer 62

depressed show reduced positive affects and arousal following gains
sig weaker response to gains in nucleas accumbens and caudate bilaterally
anhedonia and depression severity assoc with reduces caudate volume
- BG in MDD may affect the consummatory phase of reward processing