Midterm 2 Flashcards
where is the signal detection theory stemmed from?
radar activation (world war II)
what does the signal detection theory state?
that nearly ALL reasoning and decision making takes place in the presence of some uncertainty
what is the purpose of signal detection theory?
provides a precise language and graphic notation for analyzing decision making in the presence of uncertainty
what are some direct applications of signal detection theory?
- sensory experiments –> ex: outside events detected by nervous system
- medicine & diagnosis –> ex: have disease or not
- electronics & telecommunication –> ex: radars
- legal settings –> ex: someone guilty or innocent
- alarm management –> ex: emergency occurring or not
- inferential statistics & hypothesis testing –> ex: assume uncertainty has an effect but still form hypothesis and conduct experiment, just need to see if results are significant or not
what is the signal detection theory framework?
- how to look at the situation in question
- table –> 4 squares total
- vertical = ground truth (present or absent) –> not known to us
- horizontal = decision (yes or no) –> known through physical evidence
what is the purpose of the signal detection theory framework?
to maximize correct decisions (hit and CR) as much as possible while still acknowledging that errors may be present
describe the 4 outcomes of signal detection theory framework?
- hit –> ground truth = present; decision = yes (correct)
- miss –> ground truth = present; decision = no (incorrect)
- false alarm –> ground truth = absent; decision = yes (incorrect)
- correct reject –> ground truth = absent; decision = no (correct)
what was the signal detection theory framework example explained in class?
radiologist examining a CT scan
- presence or absence of a tumor
- radiologist say yes or no for tumor
what are the 2 main components of the decision-making process?
1) information acquisition
2) criterion
what is information acquisition?
- capturing all relevant empirical evident (aka information) to try to make the best decision possible
- uniformly leads to more correct decisions (hit & CR)
what will improve information acquisition
more evidence = less noise = better –> each decision is based on information from empirical evidence, but there can still be an attempt to get more or better evidence
ways to improve information acquisition and real life examples
- using better measurement devices –> ex: CT scans show shape, brightness, and texture of healthy tissue vs. tumors
- more training and practice to learn more information –> ex: doctors will know how to read a CT scan so they can use that tool to help them make their decision
- running another test –> ex: can try a better resolution test (MRI compared to CT) or can try test from a different angle
what is another benefit (that relates to midterm 1 content) of acquiring more information?
more information = more accuracy
what is criterion
- same information, same expertise, same test BUT different decisions –> based on individual’s criterion, not type of information given
- leads to a trade-off bw correct decisions (hits & CR)
what is an example of criterion
doctor isn’t sure if there is a tumor or not –> some are more likely to say yes (resulting in more false alarms), or some are more likely to say no (resulting in more misses)
criterion also includes when decisions are made based on… ?
what kind of error is perceived as more or less acceptable –> ex: some doctors may think its acceptable to have more false alarms compared to misses, or vice versa
impact of criterion: criterion shift results in more “yes”
more hits (less misses) but also more false alarms (less correct rejects)
impact of criterion: criterion shift results in more “no”
more correct reject (less false alarms) but also more misses (less hits)
what do the 2 components of the decision process (information acquisition and criterion) impact in terms of decisions?
accuracy of decisions
___ signal and ___ information uniformly leads to more correct decisions (hits or correct rejects)
greater; better
what is the trade off of criterion changes?
- they have opposing influence on the 2 incorrect decisions –> decreasing misses increases false alarms, decreasing false alarms increases misses
- more evidence not always better
what is internal response?
the variable that forms the basis of the observer’s decision
what contributed to internal response?
internal response = signal + noise
internal response takes on values that ___ from one occasion to another for the ___ same stimulus (aka “signal”)
vary; same
what are frequency of occurrence curves?
internal response (x-axis), frequency (y-axis) –> all possible realizations about a topic and how often they occur
what is the central tendency of a frequency of occurrence curve?
value that is the most likely possibility for a realization –> spread of the curve indicates that error is present
what is the criterion of a frequency of occurrence curve?
indicates in what situations to say “yes” (right from criterion line) and “no” (left from criterion line)
what is optical criterion?
the criterion that leads to the optimal amount of correct decisions based on the situation in question
criterion shift
moving the criterion line along the frequency occurrence curve(s) –> moving right = more conservative (less yes’s); moving left = less conservative (more yes’s)
hit + miss = ___
100%
false alarm + correct reject = ___
100%
what should you consider when choosing criterion?
which criterion optimizes accuracy
what does a criterion shift do?
changes the percentage of hits and false alarms of the decision –> moving right = less hits, even less false alarms; moving left = more hits, more false (but not as many as hits)
what is accuracy defined as in class in terms of frequency of occurrence curves?
how many correct answers gotten
how do you change percentage to frequency?
percentage (%) / 100 = frequency –> usually a decimal
what information do you need in order to calculate the accuracy of a situation?
- frequency (often percentage) of hits and false alarms in the population
- probability of the situation being present or absent in the population
how can the same criterion under the same noise and signal distributions results in different total accuracy?
accuracy itself depends on base rates (how the cases for the situation is distributed) –> criterion in terms of accuracy isn’t optimized until you know the base rates for that event
- base rates >50/50 (very variable) –> change criterion to be very conservative
- base rates ~50/50 –> different criterion would be represented at particular accuracy level
- base rates <50/50 –> different accuracy represented overall
how can you increase your calculated accuracy?
by shifting criterion (more or less conservative)
are some criterion values better than others?
yes, some criterion values can be better depending on…
- the outcome you are optimizing (better to maximize accuracy or minimize cost, etc)
- proportion of “signal present” and “signal absent” cases
choice of criterion: sensory experiment
- observer views a single interval (noise or signal embedded in noise) –> each event equally likely
- yes/no task –> whether or not they think signal happened
what does “populate the table” mean
fill our the percentages of hit/miss & FA/CR in the framework table
calculation of accuracy
accuracy = [ (number-of-cases-of-signal-present)(frequency-of-hits-in-decimals) + (number-of-cases-of-signal-absent)(frequency-of-correct-rejects) ] / [total-number-of-cases]
conservative
need more evidence to be able to say “yes”
liberal
say “yes”, even with just a little evidence present
what is the accuracy if you randomly tossed a coin to make your “yes/no” decision?
the accuracy you would get if you didn’t even consider empirical evidence –> not very good
how to determine optical criterion of a situation?
- sketch graph –> x axis = criterion (liberal to conservative); y axis = what trying to maximize
- find the peak of the graph (is optimal criterion)
choice of criterion: a special case
in the special case where signal absent (N) and signal present (S+N) are equally likely (base rates are 50/50), the optimal criterion that maximizes accuracy is the point where the two internal response curves cross (exact middle of the 2 distributions)
choice of criterion: signal present and absent not equally likely
- usually more representative of what happens in real life
- everything (curves, % of hits/CR, etc) all same –> just base rate diff)
- calculate accuracy the same way as before (just different number of present and absent cases –> not 50/50)
- determine optimal criterion in same way as with 50/50 base rates
why does the optimal criterion change for base rates that aren’t 50/50?
event less (or more) likely –> criterion needs to be more (or less) conservative
why is the accuracy peak higher in base rates that aren’t 50/50?
have a piece of info that creates a “pedestal” at the beginning –> results in increase accuracy
choice of criterion: optimize different parameter
- same thing to determine optimal criterion
- populate table
- calculate what trying to maximize (ex: cost)
- sketch graph
- determine peak of graph
calculation for total cost (ex of optimizing different parameter)
total cost = [ (number-of-cases-present)(percent-of-outcome-in-question-in-decimals)(cost-of-outcome-in-question-in-decimals) + (number-of-cases-absent)(percent-of-other-outcome-in-question-in-decimals)(cost-of-outcome-in-question-in-decimals) ]
what was the optimal criterion for cost in the example in class?
between C2 and C3 –> where the costs were equal
what does optimal criterion depend on?
- base rates (likelihood of proportion of “signal present” and “signal absent” cases)
- what outcome trying to maximize –> ex: accuracy, cost
discriminability
error can be minimized by reducing overlap bw the two curves
why is there always error (based on distribution curves)
because of the overlap bw the two curves
how to reduce overlap between the two curves?
- increase separation bw the curves –> get more info/evidence
- reduce the spread of the curves
discriminability index
the signal is highly discriminable from the noise when there is a large separation and a small spread
what does the discriminability index d-prime (d’) capture?
both separation and spread
calculating d’?
d’ = separation / spread
