Final - Decision Making and Testing (Streiner)

Question 1

What are the 3 statistical determinants of decision quality?

Answer 1

• validity of tests
• base rate of relevant characteristic
• selection rate

Question 2

What is the base rate?

Answer 2

• what proportion of pop being examined actually possesses the relevant characteristic

Question 3

What is the selection rate?

Answer 3

• what percentage of population will be selected
• determined in part by past use of test, success of past predictions, or # of positions available

Question 4

Define:
- cutting score:
- hits:
- misses:

Answer 4

• cutting score: score used to divide test scores into predictions ab who has characteristic vs does not
• hits: correct predictions
• misses: incorrect predictions

Question 5

Define:
- true positive:
- true negative:
- false positive:
- false negative:

Answer 5

• true positive: successfully predicted hit where they do have the characteristic
• true negative: successfully predicted hit where they don’t
• false positive: miss where characteristic is predicted but doesn’t occur (type I error)
• false negative: miss where characteristic is not predicted but does occur (type II error)

Question 6

What is the formula to get the total hit rate?

Answer 6

(true pos + true neg)/N

Question 7

What is test sensitivity? What is the formula?

Answer 7

• proportion of ppl w characteristic who are correctly detected by test
• sensitivity = true pos/(true pos + false neg)

Question 8

What is test specificity? What is the formula?

Answer 8

• proportion of ppl w/o characteristic who are correctly detected by test
• specificity = true neg/(true neg + false pos)

Question 9

When SR=100, what is the total hit rate equivalent to?

Answer 9

base rate (BR)

Question 10

When base rate is higher than selection rate, we will always make ___ decisions

Answer 10

false negative

Question 11

When SR=100, test sensitivity is ___ and specificity is ___

Answer 11

• sensitivity is 100%
• specificity is 0%

Question 12

When SR=0, test sensitivity is ___ and specificity is ___

Answer 12

• sensitivity is 0%
• specificity is 100%

Question 13

Relative to SR=0 strategy, random selection (SR=50) leads to (increased/decreased) test sensitivity and (increased/decreased) test specificity

Answer 13

• increased sensitivity
• decreased specificity

Question 14

What is the formula for the % of true positives P(TP)?

Answer 14

P(TP) = BR x SR

Question 15

What is the formula for the % of false positives P(FP)?

Answer 15

P(FP) = SR - P(TP)

Question 16

What is the formula for the % of false negatives (PFN)?

Answer 16

P(FN) = BR - P(TP)

Question 17

What is the formula for the % of true negatives P(TN)?

Answer 17

P(TN) = 100 - P(TP) - P(FP) - P(FN)

Question 18

P(TP) decisions (increase/decrease) when validity test is used

Answer 18

INCREASE (how much depends on validity of test)

Question 19

What is the formula for P(TP) when we use a validity test?

Answer 19

P(TP) = BR x SR + rxy√BR(1-BR)SR(1-SR)

Question 20

How can incremental validity be calculated for using test vs not using test to calculate P(TP)?

Answer 20

incremental validity = % hits w test - % hits w/o test

Question 21

For a high validity test, incremental validity is about __%

Answer 21

3%

Question 22

When we have a HIGH BR and high validity test, what errors are committed with vs without the test?

Answer 22

• no test: less false neg but more false pos
• test: more true negatives

Question 23

When we have a LOW BR and high validity test, what errors are committed with vs without the test?

Answer 23

• no test: less false pos but more false neg
• test: more true positives

Question 24

If a test has lower validity, its best to make decisions based on ____

Answer 24

base rates!

Question 25

Tests work best when BR = ___

Answer 25

50%

Question 26

When BR is very high, it is best to assume that ____

Answer 26

everyone has characteristic (SR=100)

Question 27

When BR is very low, it is best to assume that ____

Answer 27

no one has characteristic (SR=100)

Question 28

When in doubt about test validity, it is best to make a prediction based on _____

Answer 28

base rate

Question 29

Where can we get base rate information?

Answer 29

• already available info
• review of historical records
• build database by collecting outcome info over time