Final - Decision Making and Testing (Streiner) Flashcards
What are the 3 statistical determinants of decision quality?
- validity of tests
- base rate of relevant characteristic
- selection rate
What is the base rate?
- what proportion of pop being examined actually possesses the relevant characteristic
What is the selection rate?
- what percentage of population will be selected
- determined in part by past use of test, success of past predictions, or # of positions available
Define:
- cutting score:
- hits:
- misses:
- cutting score: score used to divide test scores into predictions ab who has characteristic vs does not
- hits: correct predictions
- misses: incorrect predictions
Define:
- true positive:
- true negative:
- false positive:
- false negative:
- true positive: successfully predicted hit where they do have the characteristic
- true negative: successfully predicted hit where they don’t
- false positive: miss where characteristic is predicted but doesn’t occur (type I error)
- false negative: miss where characteristic is not predicted but does occur (type II error)
What is the formula to get the total hit rate?
(true pos + true neg)/N
What is test sensitivity? What is the formula?
- proportion of ppl w characteristic who are correctly detected by test
- sensitivity = true pos/(true pos + false neg)
What is test specificity? What is the formula?
- proportion of ppl w/o characteristic who are correctly detected by test
- specificity = true neg/(true neg + false pos)
When SR=100, what is the total hit rate equivalent to?
base rate (BR)
When base rate is higher than selection rate, we will always make ___ decisions
false negative
When SR=100, test sensitivity is ___ and specificity is ___
- sensitivity is 100%
- specificity is 0%
When SR=0, test sensitivity is ___ and specificity is ___
- sensitivity is 0%
- specificity is 100%
Relative to SR=0 strategy, random selection (SR=50) leads to (increased/decreased) test sensitivity and (increased/decreased) test specificity
- increased sensitivity
- decreased specificity
What is the formula for the % of true positives P(TP)?
P(TP) = BR x SR
What is the formula for the % of false positives P(FP)?
P(FP) = SR - P(TP)
What is the formula for the % of false negatives (PFN)?
P(FN) = BR - P(TP)
What is the formula for the % of true negatives P(TN)?
P(TN) = 100 - P(TP) - P(FP) - P(FN)
P(TP) decisions (increase/decrease) when validity test is used
INCREASE (how much depends on validity of test)
What is the formula for P(TP) when we use a validity test?
P(TP) = BR x SR + rxy√BR(1-BR)SR(1-SR)
How can incremental validity be calculated for using test vs not using test to calculate P(TP)?
incremental validity = % hits w test - % hits w/o test
For a high validity test, incremental validity is about __%
3%
When we have a HIGH BR and high validity test, what errors are committed with vs without the test?
- no test: less false neg but more false pos
- test: more true negatives
When we have a LOW BR and high validity test, what errors are committed with vs without the test?
- no test: less false pos but more false neg
- test: more true positives
If a test has lower validity, its best to make decisions based on ____
base rates!
Tests work best when BR = ___
50%
When BR is very high, it is best to assume that ____
everyone has characteristic (SR=100)
When BR is very low, it is best to assume that ____
no one has characteristic (SR=100)
When in doubt about test validity, it is best to make a prediction based on _____
base rate
Where can we get base rate information?
- already available info
- review of historical records
- build database by collecting outcome info over time
