L20 - Prediction & Dopamine I

Question 1

Why are predictions important in the navigation of our environment?

Answer 1

• Our ability to navigate our environment is made possible by the models we create through experience
• Within the models is a set of predictions which allow us to see what happens next
• We create landmarks of areas so we know what to expect or what to associate with the place, what the place contains/its role or the purpose that it has - e.g. what type of foods at the cafe
• Our predictions create our cognitive models - if predictions are not accurate = models not accurate
• Main goal of reinforcement learning is to make sure predictions match reality

Question 2

How can prediction error help us update our models of the world?

Answer 2

• We develop models of the things that matter to us
• When we make an error in our prediction we need to know we have made an error
• We need to update our models of environment with what actually happened against what we expected to happen - so we can create a better model to learn about the new thing in our environment
• When something unexpected happens in our environment, we are surprised
• Mathematically → Surprise occurs when your prediction does not match reality
• Don’t learn in one episode, learn over a series of episodes - need to know that it isn’t just a one off event before we incorporate it into our model
• Associative strength - as this gets stronger, our prediction error gets smaller as out learning is increasing
• Learning episode 1 - Were not expecting gluten free pizza. Below shows the discrepancy between what you thought was going to happen and what did.
  
  • It shows that associative strength increases each time (gradual learning)

Question 3

Why is prediction error a necessary catalyst for learning?

Answer 3

• If something that doesn’t tell us that we’re wrong, we won’t know that we are wrong
• ********Blocking phenomenon**********
  
  • Use aversive outcome (so negative) and also positive (reward based)
    1. Come into experiment, sit in front of computer, complete 50 trials and get shocked when given purple stimulus
    
    1. Prediction errors occur at the first time you get shocked at you were not expecting it. Don’t have prediction error by the end of it
       2. For 20 trials, presented with multi-coloured stimulus at the same time as the purple stimulus when shock given. PPT not surprised. Purple stimulus blocks out what the multicoloured stimulus could mean to the participant as only focussing on getting shocked
       3. Then just shown the multicoloured stimulus - PPT shocked, if PPT not surprised so shows blocking
  **This is the same as classical conditioning or associative learning so introduce a control condition alongside to try and mediate this*
  1. Have mussel stimulus intermingled with purple stimulus when the 50 trials occurs but there is no shock given with it
  2. Then mussel is paired with orange symbol which shocks (again intermingled with orange)
  3. Can then compare if they expect shock with orange stimulus to see if blocking works - so blocking.
• Expect them to make predictions they are going to get shocked when see purple stimulus but not mussel
• Asked how afraid they are and also measure on physiological response
  
  • They are making prediction errors early on but this becomes lower over time
    
    • Looked at physiological responses such as skin conductance and heart rates is higher when expecting purple stimulus
• After they learnt that the purple stimulus links to the mussel stimulus, does it change what is learnt in the second phase
• Compare multicoloured stimulus vs orange stimulus by themselves
• Found they are expecting to get shocked for orange stimulus so demonstrates the blocking process of stimuli we don’t need to be concerned about. Multicoloured stimulus has been learnt about less as the purple stimulus had been paired with purple.

So if you have experience with a stimulus and an outcome and another stimulus comes in a does the same thing, learning will be blocked as they don’t have prediction error (as they are not suprised)

Question 4

As a summary, why is prediction error a necessary catalyst for learning?

Answer 4

• The blocking procedure shows that we only learn about associations when there is a prediction error.
• Prediction error is the necessary catalyst for learning.
• Prediction error tells us when we’re wrong so we can update our predictions and build better models of our environment.

Question 5

How was the same experiment used but looking at reward rather than punishment with dopamine neurons?

Answer 5

• Same task as before but paired with reward rather than shock
• Tobler et al., 2006
  
  • If we’re looking at BOLD activity in the brain, the most important part of the task to see if prediction errors are different is during the compound part
  • Expecting prediction errors in the experimental condition but not in the control condition so can look if there is an area in the brain that also differs in these contexts to see what might be responsible for prediction errors
  • Yellow activity shows there is a change in activity across learning
• There is a greater magnitude of change in activity in the control trials relative to the experimental trials - It is in the dopamine area
• Change in activity could be excitation or inhibition, we just know it is a region of interest
• Between the brain regions that are important for prediction errors compared to the brain regions that are important for storing maps
  
  • Prediction errors help you learn - tell other areas of the brain to learn, it is just the surprise element
  • Tested memory - Where is the model?
• Know that orange this is related to juice reward and that multi-coloured isn’t associated with it
• Memory of it is a consequence of prediction errors but are not the prediction error itself
• Surprise signals in midbrain, but learned associations occurred in the orbitofrontal cortex
  
  • Activity in the orbitofrontal cortex during the test is correlated with effective blocking
  • If an individual is good at blocking, more activity in orbitofrontal cortex