Expert Judgement

Question 1

Judgements are made at a _____ level and a _____

level

Answer 1

Judgements are made at a qualitative level and a quantitative level.

Question 2

In “Structured Expert Judgement”, the decision maker seeks to adopt uncertainties provided by experts, however there are some issues:
•
•
•

Answer 2

• Each expert will probably think differently
• We may need to combine different assessments in a reasonable way
• Some experts will be better than others, fortunately this is testable.

Question 3

Name some features of Structured Expert.

Answer 3

• Identification
• Role Separation
• Equality
• Eliminating Ambiguity
• Training
• Neutrality
• Transparency
• Accountability
• Rationales
• Empirical Control

Question 4

What is the Delphi Method?

Answer 4

This method works by inviting experts to make predictions.
Those outside the “central tendency” are asked if they want to revise their opinions and must justify staying outside the central tendency.
However, there are a number of shortcomings in this method which means that it isn’t always applicable.

Question 5

What are the 6 Biases which the Delphi Method suffers from?

Answer 5

Structural Bias
Motivational Bias
Cognitive Bias
Overconfidence
Anchoring
Availability

Question 6

Structural bias is about being influenced by the ____ in which the system under study is ____ .

Answer 6

Structural Biases is about being influenced by the form in which the system under study is presented.

Question 7

Motivational bias is about responding to _____, which might be ______.

Answer 7

Motivational bias is about responding to rewards or penalties, which might be explicit or implicit.

Question 8

Cognitive Bias is ______ because of the expectations of the expert.

Answer 8

Cognitive Bias is interpreting the question in a particular way because of the expectations of the expert.

Question 9

Overconfidence is when _____________, and find it difficult to admit that they cannot ___________.

Answer 9

Overconfidence is when experts underestimate their own lack of knowledge, and find it difficult to admit that they cannot predict everything with confidence.

Question 10

Anchoring is the tendency of an individual to __________.

Answer 10

Anchoring is the tendency of an individual to rely too heavily on an initial piece of information.

Question 11

Availability is when the rate of commonly occurring events are _________.

Answer 11

Availability is when the rate of commonly occurring events are underestimated.
Conversely, uncommon events are overestimated.

Question 12

What are the 7 steps of the Stanford Research Institutes guide on developing a “Process of Elicitation”

Answer 12

1. Motivate
2. Structure
3. Condition
4. Encode
5. Verify
6. Aggregate
7. Discretise

Question 13

For “normalised geometric” methods we need to weigh the expert’s opinions.
• Why?
• What can we do to address this?

Answer 13

• We can’t just give everyone the same weight because if there’s too many experts from the same school then their weight will become disproportionately large.
• Instead we use a “scoring rule” where experts are tested and awarded more weight if they can justify and prove it’s their own opinions.

Question 14

Cooke identified several principles which should underlie the choice of expert judgement techniques, they are…?
(5 Available)

Answer 14

• Reproducibility - All calculations should be reproducible.
• Accountability - Identify the source of expert subjective probabilities.
• Neutrality - The method encourages experts to state true opinions.
• Fairness - All experts are treated equally a-priori.
• Empirical Control - Expert assessments should be susceptible in principle to empirical control.

Question 15

What are paired comparison models?

Answer 15

It is a model which compares two entities to decide which one is preferred or possesses the greater amount of quantitative property.

Question 16

Probabilities can be identified using _____ factors, but it is necessary to ______ these methods by assessing two previously known values.

Answer 16

Probabilities can be identified using scaling factors, but it is necessary to calibrate these methods by assessing two previously known values.

Question 17

Briefly describe the Thurstone Model

Answer 17

Assumes that V(i,e) is normally distributed.

It commonly assumes that:
• The mean value is the “true value” that we are trying to assess,
• The standard deviations of the experts are all constant for each value

Each expert is asked about a pair or values, 𝑖, 𝑗 and which is more likely.

Question 18

Briefly describe the Bradley-Terry (Failure Rate) Model

Answer 18

Formulated for assessing constant failure rates of equipment. The statement, “i is preferred to j” would be interpreted as “component i will fail after component j”.

Question 19

What similarities exist between Thurstone and the Bradley-Terry models?

Answer 19

Both models are paired comparison models, that is they require experts to give preferences between pairs of alternatives.
i.e. input data is qualitative rather than quantitative.
They make rather strong and untestable assumptions about the structure of the experts’ knowledge.

Question 20

Briefly describe the Apostolakis-Mosleh Bayesian Model?

Answer 20

An example of the Bayesian approach to expert judgement.

Assumes that when assessing a quantity whose true value is 𝑥, each expert will give an answer of 𝑋1=𝑥+𝜀1.
• The mean is a measurement of the bias of the expert
• The standard deviation is a measurement of the degree of accuracy

It is usual for the analyst to assess the expert means and standard deviations, in addition to supply the prior distribution.

Question 21

Briefly describe Cooke’s Classical Method

Answer 21

Loosely satisfies all five expert judgement requirements (Reproducibility, Accountability, Empirical Control, Neutrality, and Fairness).

Uses 2 components,
• Calibration shows whether experts are assessing probability accurately
• Information shows whether experts give narrow or broad uncertainty bands

Question 22

There are many possible ways to combine probabilities, but two methods which are in common use are:
•
•

Answer 22

There are many possible ways to combine probabilities, but two methods which are in common use are:
• Normalised Geometric means
• Arithmetic means

Question 23

Describe “Normalised Geometric means” for probability judgement

Answer 23

When expert 𝑗 gives probabilities 𝑃1𝑗,…𝑃𝑛𝑗 we take the weighted geometric mean of each of the events and then normalise to get the actual probability

Question 24

Describe “Arithmetic means” for probability judgement

Answer 24

This is when we take combined arithmetic probabilities to be the weighted arithmetic mean

Question 25

State the problem / limitation of the Normalised Geometric and Arithmetical means for probability judgement
(1 each)

Answer 25

• The independence property is only satisfied by the geometric averaging rule
• The marginalisation is satisfied by the arithmetical averaging rule.

Therefore we have to make a choice for which rule to use!

Question 26

What is the Sheffield Elicitation Framework (SHELF)?

Answer 26

Also known as behavioral aggregation.
Groups of experts are asked to collectively agree to a distribution that a Rational Impartial Observer (RIO) would agree to.

Some argue that this makes more sense the weighting experts and taking a mixture distribution (as recommended by the Classical model).