2. Measurement and Testing Flashcards
Give examples of early literature on measuring confidence
Fischoff, Slovic & Lichtenstein 1977
Fischoff & Lichtenstein 1981
What is systematic mis-calibration?
-people having levels of confidence which don’t match their test outcomes in a systematic way
How can confidence mis-callibration show itself?
-people being overconfident
-task dependent- some people were overconfident on hard tasks but under confident on easy tasks
Which papers provide evidence of confidence mis-calibration?
Keren 1991
Alba & Hutchinson 2000
What did more recent work on confidence mis-calibration find?
Either less mis-calibration or under confidence when there were incentivised confidence measurement tools
Why does confidence matter?
Confidence mis-calibration affects behaviour in the lab (Camerer & Lovallo 1999) and in the field (Grubb 2015) on consumer decisions
How might incentivised judgments affect confidence calibration?
They might be better calibrated but they could instead by biased by risk attitudes if people aren’t risk neutral
What were the research questions of Murad, Sefton & Starmer 2016?
Do risk attitudes generate bias in incentivised confidence measurement?
If we control for risk attitudes, do incentives make much difference?
Is an individuals confidence related to their risk attitude?
In the Murad, Sefton & Starmer 2016 study what did they compare?
-compare measurements of confidence with and without incentives
-measure risk attitudes and decontaminated incentivised confidence (assuming RDU)
-compare confidence with/without incentives and the decontaminated incentivised confidence
-finally see if there is correlation between confidence without incentives and risk attitudes
In Murad, Sefton & Starmer 2016 what confidence measurement tools did they use?
-reported confidence- answer Q then answer how confident you are that you are correct in %
-incentivised confidence- answer same Q then choose between gamble and safe option based on answer given
In Murad, Sefton & Starmer 2016 how would being risk averse affect the incentivised confidence task?
Risk averse agents will switch from sure option to gamble at a lower switch. This produces downwards bias in confidence
What are the design details in Murad, Sefton & Starmer 2016?
Part 1: risk elicitation (25 tasks)
Part 2: 20 quiz tasks with confidence elicitation
Subjects is rewarded for either part 1 or part 2
86 subjects, 40 incentivised, 46 reported
Results of Murad, Sefton & Starmer 2016
-no significant mis-calibration
-hard/easy effect in non incentivised
-under confidence in incentivised tasks
-those who are risk averse are less confident
-individuals tend to overweight small probabilities and underweight high probabilities
-when we back out confidence assuming the prospect theory model we get very similar results to the reported confidence
-lower confidence associated with underweighting of probability
What is preference reversal?
The idea that people aren’t consistent in their preferences. This is often shown in P bet and $ bet where most people choose P but value $ higher. Standard reversal
P>$
V($)>V(P)
What is the Cubitt, Munro & Starmer 2004 study about?
It tested psychological hypotheses about preference reversal whilst controlling for economic theories
How did Grether & Plott 1979 challenge preference reversal?
They argued this can be interpreted as there being different mental decision processes between choice and valuation. This behaviour isn’t explained by stable preferences
What were the key design elements in Cubitt, Munro & Starmer 2004?
Ordinal payoff scheme
Probabilistic valuation
Describe the ordinal payoff scheme in Cubitt, Munro & Starmer 2004
-subjects consider multiple P/$ pairs
-three tasks for each pair: choose P or $, value P, value $
-at the end, random choice device selects one P/$ pair- subject plays either P or $
-second random device chooses “choice” or “value”
-subject plays chosen option if “choice” and most highly valued option if “valuation”
What does OPS control for?
Generalised Economic Theories of PR
What does generalised economic theories allow?
It allows violation of expected utility theory but retains context free preferences
What are the two valuation models in Cubitt, Munro & Starmer 2004?
MV gives money valuation for each lottery, L - give m such that L~(m,1)
PV give probability valuation for each lottery, L - given p such that L~(£10,p)
From the viewpoint of GET what should we expect in the Cubitt, Munro & Starmer 2004?
The two valuation groups face identical tasks so we expect no systematic PR, no difference between groups
How does scale compatibility explain PR?
Any task has a response mode. Attributes of goods vary in compatibility with response mode. Attribute weight increases with compatibility so prize money gets more weight in monetary valuation relative to choice. This could explain standard PR
What does scale compatibility predict in Cubitt, Munro & Starmer 2004 and what are actual results?
Predicts standard PR in MV group and counter standard PR in PV group. This is in line with results, although still too much standard PR in PV
MV- 30% PR, <5% counter standard PR
PV-20% standard PR, 25% counter PR
Broad conclusions of Cubitt, Munro & Starmer 2004
Found systematic PR when controlling for economic hypotheses. Treatment effect is consistent with direction compatibility hypothesis. Compatibility isn’t the whole story- conjunctions of effects, errors, regret
How is miscalibiration bias measured in Murad, Sefton Starmer 2016?
Miscalibiration bias is average confidence minus the proportion of correct answers. They then test if the mean of the distribution is equal to zero using a t-test
How much does mean bias fall when inferred confidence is under RDU rather than EV in Murad, Sefton & Starmer 2016?
Mean bias falls from -0.212 ( inferred confidence under EV) to -0.086 (inferred confidence under RDU)
How do Murad, Sefton & Starmer 2016 elicit risk preferences?
They follow Bruhin, Fehr-Duda & Epper (2010). Consider a 50:50 chance of £10 or nothing vs a guaranteed sum of money. Choose preferred option for each row in the table.
Power utility function with two parameter probability weighting to estimate risk attitudes
P bet
Smaller payoff but more certain
(£8, 0.6)
$ bet
Higher payoff but lower certainty
(£18, 0.3)
What are context free preferences?
-preferences over lotteries are described by some choice function C(.)
-C(.) picks preferred subset from any feasible set of prospects
-same C(.) governs behaviour in all tasks
Why does GET provide no reason to expect systematic pattern of PR?
If rankings of choice and valuation differ, the subject selects the lottery (P, 0.5; $, 0.5). Note this is the same lottery for standard and non standard reversal. Implies PR may occur if subjects are indifferent between P and $ or if they are non-EU. But there is no reason to expect systematic pattern
What 3 psychological explanations are tested for in Cubitt, Munro & Starmer 2004?
Prominence
Task goal
Scale compatibility
What does prominence theory predict in Cubitt, Munro & Starmer 2004?
If prominence is the source of PR then any systematic tendency towards PR in a given parameter set should be in the same direction in both groups
