2. Measurement and Testing

Question 1

Give examples of early literature on measuring confidence

Answer 1

Fischoff, Slovic & Lichtenstein 1977
Fischoff & Lichtenstein 1981

Question 2

What is systematic mis-calibration?

Answer 2

-people having levels of confidence which don’t match their test outcomes in a systematic way

Question 3

How can confidence mis-callibration show itself?

Answer 3

-people being overconfident
-task dependent- some people were overconfident on hard tasks but under confident on easy tasks

Question 4

Which papers provide evidence of confidence mis-calibration?

Answer 4

Keren 1991
Alba & Hutchinson 2000

Question 5

What did more recent work on confidence mis-calibration find?

Answer 5

Either less mis-calibration or under confidence when there were incentivised confidence measurement tools

Question 6

Why does confidence matter?

Answer 6

Confidence mis-calibration affects behaviour in the lab (Camerer & Lovallo 1999) and in the field (Grubb 2015) on consumer decisions

Question 7

How might incentivised judgments affect confidence calibration?

Answer 7

They might be better calibrated but they could instead by biased by risk attitudes if people aren’t risk neutral

Question 8

What were the research questions of Murad, Sefton & Starmer 2016?

Answer 8

Do risk attitudes generate bias in incentivised confidence measurement?
If we control for risk attitudes, do incentives make much difference?
Is an individuals confidence related to their risk attitude?

Question 9

In the Murad, Sefton & Starmer 2016 study what did they compare?

Answer 9

-compare measurements of confidence with and without incentives
-measure risk attitudes and decontaminated incentivised confidence (assuming RDU)
-compare confidence with/without incentives and the decontaminated incentivised confidence
-finally see if there is correlation between confidence without incentives and risk attitudes

Question 10

In Murad, Sefton & Starmer 2016 what confidence measurement tools did they use?

Answer 10

-reported confidence- answer Q then answer how confident you are that you are correct in %
-incentivised confidence- answer same Q then choose between gamble and safe option based on answer given

Question 11

In Murad, Sefton & Starmer 2016 how would being risk averse affect the incentivised confidence task?

Answer 11

Risk averse agents will switch from sure option to gamble at a lower switch. This produces downwards bias in confidence

Question 12

What are the design details in Murad, Sefton & Starmer 2016?

Answer 12

Part 1: risk elicitation (25 tasks)
Part 2: 20 quiz tasks with confidence elicitation
Subjects is rewarded for either part 1 or part 2
86 subjects, 40 incentivised, 46 reported

Question 13

Results of Murad, Sefton & Starmer 2016

Answer 13

-no significant mis-calibration
-hard/easy effect in non incentivised
-under confidence in incentivised tasks
-those who are risk averse are less confident
-individuals tend to overweight small probabilities and underweight high probabilities
-when we back out confidence assuming the prospect theory model we get very similar results to the reported confidence
-lower confidence associated with underweighting of probability

Question 14

What is preference reversal?

Answer 14

The idea that people aren’t consistent in their preferences. This is often shown in P bet and $ bet where most people choose P but value $ higher. Standard reversal
P>$
V($)>V(P)

Question 15

What is the Cubitt, Munro & Starmer 2004 study about?

Answer 15

It tested psychological hypotheses about preference reversal whilst controlling for economic theories

Question 16

How did Grether & Plott 1979 challenge preference reversal?

Answer 16

They argued this can be interpreted as there being different mental decision processes between choice and valuation. This behaviour isn’t explained by stable preferences

Question 17

What were the key design elements in Cubitt, Munro & Starmer 2004?

Answer 17

Ordinal payoff scheme
Probabilistic valuation

Question 18

Describe the ordinal payoff scheme in Cubitt, Munro & Starmer 2004

Answer 18

-subjects consider multiple P/$ pairs
-three tasks for each pair: choose P or $, value P, value $
-at the end, random choice device selects one P/$ pair- subject plays either P or $
-second random device chooses “choice” or “value”
-subject plays chosen option if “choice” and most highly valued option if “valuation”

Question 19

What does OPS control for?

Answer 19

Generalised Economic Theories of PR

Question 20

What does generalised economic theories allow?

Answer 20

It allows violation of expected utility theory but retains context free preferences

Question 21

What are the two valuation models in Cubitt, Munro & Starmer 2004?

Answer 21

MV gives money valuation for each lottery, L - give m such that L~(m,1)
PV give probability valuation for each lottery, L - given p such that L~(£10,p)

Question 22

From the viewpoint of GET what should we expect in the Cubitt, Munro & Starmer 2004?

Answer 22

The two valuation groups face identical tasks so we expect no systematic PR, no difference between groups

Question 23

How does scale compatibility explain PR?

Answer 23

Any task has a response mode. Attributes of goods vary in compatibility with response mode. Attribute weight increases with compatibility so prize money gets more weight in monetary valuation relative to choice. This could explain standard PR

Question 24

What does scale compatibility predict in Cubitt, Munro & Starmer 2004 and what are actual results?

Answer 24

Predicts standard PR in MV group and counter standard PR in PV group. This is in line with results, although still too much standard PR in PV
MV- 30% PR, <5% counter standard PR
PV-20% standard PR, 25% counter PR

Question 25

Broad conclusions of Cubitt, Munro & Starmer 2004

Answer 25

Found systematic PR when controlling for economic hypotheses. Treatment effect is consistent with direction compatibility hypothesis. Compatibility isn’t the whole story- conjunctions of effects, errors, regret

Question 26

How is miscalibiration bias measured in Murad, Sefton Starmer 2016?

Answer 26

Miscalibiration bias is average confidence minus the proportion of correct answers. They then test if the mean of the distribution is equal to zero using a t-test

Question 27

How much does mean bias fall when inferred confidence is under RDU rather than EV in Murad, Sefton & Starmer 2016?

Answer 27

Mean bias falls from -0.212 ( inferred confidence under EV) to -0.086 (inferred confidence under RDU)

Question 28

How do Murad, Sefton & Starmer 2016 elicit risk preferences?

Answer 28

They follow Bruhin, Fehr-Duda & Epper (2010). Consider a 50:50 chance of £10 or nothing vs a guaranteed sum of money. Choose preferred option for each row in the table.

Power utility function with two parameter probability weighting to estimate risk attitudes

Question 29

P bet

Answer 29

Smaller payoff but more certain
(£8, 0.6)

Question 30

$ bet

Answer 30

Higher payoff but lower certainty
(£18, 0.3)

Question 31

What are context free preferences?

Answer 31

-preferences over lotteries are described by some choice function C(.)
-C(.) picks preferred subset from any feasible set of prospects
-same C(.) governs behaviour in all tasks

Question 32

Why does GET provide no reason to expect systematic pattern of PR?

Answer 32

If rankings of choice and valuation differ, the subject selects the lottery (P, 0.5; $, 0.5). Note this is the same lottery for standard and non standard reversal. Implies PR may occur if subjects are indifferent between P and $ or if they are non-EU. But there is no reason to expect systematic pattern

Question 33

What 3 psychological explanations are tested for in Cubitt, Munro & Starmer 2004?

Answer 33

Prominence
Task goal
Scale compatibility

Question 34

What does prominence theory predict in Cubitt, Munro & Starmer 2004?

Answer 34

If prominence is the source of PR then any systematic tendency towards PR in a given parameter set should be in the same direction in both groups