Competing hypothesis Flashcards
What is the ‘diagnosticity’ of evidence and/or arguments
Which items are most helpful in judging the relative likelihood of alternative hypotheses.
Evidence is diagnostic when it influences your judgment on the relative likelihood of the various hypotheses
If an item of evidence seems consistent with all the hypotheses, it may have no diagnostic value.
A common experience is to discover that most of the evidence supporting what you believe is the most likely hypothesis really is not very helpful, because that same evidence is also consistent with other hypotheses. When you do identify items that are highly diagnostic, these should drive your judgment.
What are this common reason for analysis to be proved wrong
When analysis turns out to be wrong, it is often because of key assumptions that went unchallenged and proved invalid
But, the problem is to determine which assumptions merit questioning.
One advantage of ACH is that it gives you a good idea of what should be rechecked
How should take analysis once it is finished and conclusions drawn
Identify milestones for future observation that may indicate events are taking a different course than expected.
Analytical conclusions are always tentative
Having prepared a matrix how should one go about analysing the diagnosticity of the evidence/arguments
Analyze how each piece of evidence relates to each hypothesis. This differs from the normal procedure, which is to look at one hypothesis at a time in order to consider how well the evidence supports that hypothesis.
At this point, Step 3, take one item of evidence at a time, then consider how consistent that evidence is with each of the hypotheses.
To fill in the matrix, take the first item of evidence and ask whether it is consistent with, inconsistent with, or irrelevant to each hypothesis. Then make a notation accordingly in the appropriate cell under each hypothesis in the matrix.
Whilst analysing diagnosticity of evidence in the matrix, what else might you want to consider at the same step
- Adding a scale to show the intrinsic importance of each item of evidence.
- Adding a scale to show the ease with which items of evidence could be concealed, manipulated, or faked, or the extent to which one party might have an incentive to do so. This may be appropriate when the possibility of denial and deception is a serious issue.
What are the three key factors which distinguish ACH from intuitive analysis
- Analysis starts with a full set of alternative possibilities, rather than with a most likely alternative for which the analyst seeks confirmation. This ensures that alternative hypotheses receive equal treatment and a fair shake.
- Analysis identifies and emphasizes the few items of evidence or assumptions that have the greatest diagnostic value in judging the relative likelihood of the alternative hypotheses. In conventional intuitive analysis, the fact that key evidence may also be consistent with alternative hypotheses is rarely considered explicitly and often ignored.
- Analysis of competing hypotheses involves seeking evidence to refute hypotheses. The most probable hypothesis is usually the one with the least evidence against it, not the one with the most evidence for it. Conventional analysis generally entails looking for evidence to confirm a favored hypothesis.
In what way should go about collecting evidence and arguments
First, list the general evidence that applies to all the hypotheses.
Then consider each hypothesis individually, listing factors that tend to support or contradict each one. You will commonly find that each hypothesis leads you to ask different questions and, therefore, to seek out somewhat different evidence.
For each hypothesis, ask yourself this question: If this hypothesis is true, what should I expect to be seeing or not seeing?
Note the absence of evidence as well as its presence. For example, when weighing the possibility of military attack by an adversary, the steps the adversary has not taken to ready his forces for attack may be more significant than the observable steps that have been taken.
How should you best identify the possible hypothesis to be considered
What should you bear in mind when screening out improbable hypothesis to reduce time wasting
How many hypotheses should be considered
Brainstorming of a wide array
When screening distinguish between disproved and unproven - for an unproven hypothesis there is no evidence it is correct, but for disproved their is positive evidence it is wrong
Unproven hypotheses should be kept going
More than seven hypotheses is probably unmanageable, but as a general rule, the greater the level of uncertainty or the greater the policy impact, the more should be considered.
What should you do after you’ve analysed the diagnostic value of evidence
Refine the matrix
Reconsider the hypotheses and delete evidence and arguments that have no diagnostic value.
You might also be better placed to reword hypotheses, or make finer distinctions or new hypotheses added.. Conversely if there is no way of distinguishing between two hypotheses perhaps they can be simplified into one
you should add other evidence which you now realise is affecting your thinking
Store deleted evidence for records
What should you do once you’ve refined your matrix
How should approach it
Draw tentative conclusions about the relative likelihood of each hypothesis
Approach it by trying to disprove the hypothesis rather than prove it
Work down the matrix, looking at each hypothesis as a whole (unlike earlier step 3 where you work across hypotheses for each piece of evidence)
In examining the matrix, look at the minuses, or whatever other notation you used to indicate evidence that may be inconsistent with a hypothesis. The hypotheses with the fewest minuses is probably the most likely one. The hypothesis with the most minuses is probably the least likely one.
This initial ranking by number of minuses is only a rough ranking, however, as some evidence obviously is more important than other evidence, and degrees of inconsistency cannot be captured by a single notation such as a plus or minus. By reconsidering the exact nature of the relationship between the evidence and the hypotheses, you will be able to judge how much weight to give it.
What should you consider if you are tempted to write ‘there is no evidence that…’
If this hypothesis is true, can I realistically expect to see evidence of it?
E.g. if there concealment was a key component of the plan
What should you do if you disagree with the matrix findings that a given hypothesis is probable or unlikely
If so, it is because you omitted from the matrix one or more factors that have an important influence on your thinking. Go back and put them in, so that the analysis reflects your best judgment.
This procedure forces you to spend more analytical time than you otherwise would on what you had thought were the less likely hypotheses.
What should you do once you’ve got your rough analysis of the matrix
Test sensitivity of conclusions to critical items of evidence
How do you go about testing the sensitivity of your conclusion to pieces of evidence
In Step 3 you identified the evidence and arguments that were most diagnostic, and in Step 5 you used these findings to make tentative judgments about the hypotheses.
Now, go back and question the few linchpin assumptions or items of evidence that really drive the outcome of your analysis in one direction or the other.
- Are there questionable assumptions that underlie your understanding and interpretation?
- Are there alternative explanations or interpretations?
- Could the evidence be incomplete and, therefore, misleading?
This is a good time to look at concerns over denial and deception
How should you consider the evidence and arguments to be considered
Broadly
They refer to all the factors that have an impact on your judgments about the hypotheses, and should not be limited to concrete evidence
Also include your own assumptions or logical deductions about another person’s or group’s or country’s intentions, goals, or standard procedures
