Extreme Psychometrics Flashcards
Statement 1: Someone who has a conservative response bias in a recognition memory task will be less likely to remember the task items than someone who has a liberal response bias.
Statement 2: When scoring memory accuracy in a recognition memory task, we need to measure both the correct hit rate and the false positive rate.
(a) Both statements are true.
(b) Statement 1 true; Statement 2 false.
(c) Statement 1 false; Statement 2 true.
(d) Both statements are false.
The answer was c. See Lecture 10. Statement 1 is false - in this context, the level of memory would be measured by sensitivity not response bias. That is, differences in response bias would not be expected to affect level of memory. Statement 2 is true. We need both correct hit rate and false positive rate to work out the level of memory. If, for example, we only measured correct hit rate, then our measurement of memory (sensitivity) would be confounded by response bias.
Statement 1: Signal Detection Theory was originally developed for use in recognition memory tasks.
Statement 2: Recognition memory tasks involve people making discrimination judgements.
(a) Both statements are true.
(b) Statement 1 true; Statement 2 false.
(c) Statement 1 false; Statement 2 true.
(d) Both statements are false.
The answer was c. See Lecture 10. Statement 1 is false. Signal Detection Theory was originally developed for use in perception experiments not recognition memory tasks. Statement 2 is true. Recognition memory tasks involve test takers attempting to discriminate between stimuli (e.g. words) they have seen before and stimuli they have not seen before.
Imagine you are given a test of hearing where you are played a series of beeps of different volumes over headphones. Some of these beeps are too quiet to hear: the test is designed to find the quietest volume you can hear. Your task is to indicate when you can hear a tone. If you want to try and pretend your hearing is better than it might be and you know that the test is NOT employing signal detection theory, what should you do?
(a) Adopt a liberal criterion for responding.
(b) Adopt a conservative criterion for responding.
(c) Indicate you can hear beeps only when you are certain you can hear them.
(d) Only indicate you can hear beeps when you definitely can’t hear them.
The answer was a. See Lecture 10, slides entitled “Signal Detection Theory”. The described task is analogous to the recognition memory task – where the “signal” is the tone and “noise” is no tone. That is, you are trying to discriminate between the presence or absence of a tone. A liberal criterion (saying “yes” even when you have doubts that a tone was presented) would maximize correct hits.
Statement 1: Signal Detection Theory is useful when investigating jurors’ judgements of the innocence or guilt of a defendant.
Statement 2: For quality control inspectors in a factory, the rate of detecting faulty products decreased over the duration of a shift (Davies & Parasuraman, 1982). This was found to be due to a change in their sensitivity rather than their response bias.
(a) Both statements are true.
(b) Statement 1 true; Statement 2 false.
(c) Statement 1 false; Statement 2 true.
(d) Both statements are false.
The answer was b. See Lecture 10. Statement 1 is true. Jurors have to discriminate between innocent and guilty defendants. Hence it’s a discrimination task and hence Signal Detection Theory can be applied. Statement 2 is false. Davies and Parasuraman (1982) found that the decrease in rate of detecting faulty products over a shift was due to changes in response bias not sensitivity.
What are item characteristic curves?
(a) A plot of ability versus item discrimination index.
(b) A plot of ability versus item difficulty index.
(c) A plot of ability versus the item-total correlation.
(d) A plot of ability versus the item validity coefficient.
The answer was b. See Lecture 10. Item characteristic curves are a plot of level of ability (horizontal axis) versus the chance of people getting the item correct - i.e. the item difficulty index (vertical axis).
Which of the following is NOT a key advantage of Item Response Theory (over Classical Test Theory)?
(a) You can use sophisticated linear correlation coefficients to model performance.
(b) Item Response Theory is better for computerised adaptive testing.
(c) You do not need to give all the items in a test to calculate someone’s score.
(d) You can compare the ability of two people in a test even if they have not completed the same test items.
The answer was a. See Lecture 10. Option a is NOT a key advantage, because Item Response Theory uses non-linear models (e.g. those curved s-shaped item characteristic curves). All the other options are examples of advantages of Item Response Theory over Classical Test Theory.
In the three parameter model used in Item Response Theory, which of the following is NOT one of the three parameters used to estimate the probability of getting a particular question correct for a certain level of ability?
(a) Discrimination.
(b) Difficulty.
(c) Trait score.
(d) Level of guessing.
The answer was c. See Lecture 10. The three parameters in the three parameter model are discrimination, difficulty, and level of guessing.
Statement 1: In Signal Detection Theory, d’ (d prime) is the difference between the signal and noise distributions in standard deviations.
Statement 2: If d’ (d prime) is zero then the signal and noise cannot be distinguished. (a) Both statements are true. (b) Statement 1 true; Statement 2 false. (c) Statement 1 false; Statement 2 true. (d) Both statements are false.
The answer was a. See Lecture 10. Statement 1 is true - d’ is indeed the difference between signal and noise distributions in standard deviation units. Statement 2 is true. If d’ is zero, there’s no difference between the signal and noise distributions and hence they cannot be distinguished. In the context of the recognition memory test, it means that people cannot distinguish words they have seen before from words they have not (i.e. they’ve completely lost their memory for the words).
Statement 1: In Item Response Theory, Theta is a measure of a test taker’s overall level of ability (assuming they’re completing some sort of achievement or aptitude test).
Statement 2: When using Item Response Theory, it is possible that two people could get the same number of questions correct in some aptitude test but nonetheless end up with completely different test scores.
(a) Both statements are true.
(b) Statement 1 true; Statement 2 false.
(c) Statement 1 false; Statement 2 true.
(d) Both statements are false.
The answer was a. See Lecture 10. Statement 1 is true. Theta is a key outcome measure in item response theory - reflecting the test taker’s level of ability in the context of aptitude/achievement-type tests. Statement 2 is true. When applying Item Response Theory, the use of adaptive testing could mean that different test takers end up answering different questions. So two people could get the same number of questions correct, but the person who was answering more difficult questions would get a higher score to reflect the fact they were able to get more difficult questions correct.
Computerized adaptive testing refers to:
(a) A testing situation in which all examinees receive the same test questions in a standardized fashion.
(b) A testing situation in which test items administered may vary for each examinee on the basis of responses to prior items.
(c) A testing situation in which the computer’s task is to score responses and provide output data related to item analysis.
(d) A testing situation in which test items administered to an examinee are selected on the basis of the examinee’s demographic characteristics.
The answer was b. Computerized adaptive testing involves choosing the questions to give to a test taker on the basis of whether they got previous questions right or wrong. This allows the test to “home in” on the test taker’s ability level using fewer questions. For example, if a test taker got 5 medium-difficulty questions correct then I might choose to skip all the easier questions (because I can be confident they’re very likely to get them correct too) and instead present them with harder questions.
In a psychometric function, derived from psychophysical data in which participants are attempting to discriminate between speeds in a driving simulator, what is discrimination measured by?
(a) The area underneath the curve.
(b) The point at which the curve crosses the 50% level.
(c) The slope of the curve.
(d) The point where the curve intercepts the vertical axis.
The answer was c. See Lecture 12. In the context described, discrimination is measured by the slope of the curve. Discrimination is the area under the curve for ROC curves - but not psychometric functions based on psychophysical data.
Statement 1: The pass mark that minimizes the false positive rate of a diagnostic test is the point on the ROC curve where the sum of sensitivity and specificity is highest.
Statement 2: If the area under a ROC curve is estimated to be 0.5 for a particular diagnostic test then it means that the test has no diagnostic value. (a) Both statements are true. (b) Statement 1 true; Statement 2 false. (c) Statement 1 false; Statement 2 true. (d) Both statements are false.
The answer was c. See Lecture 12. Statement 1 is false because maximizing the sum of sensitivity and specificity will give you the best pass mark for discriminating between whatever the test was supposed to be predicting (e.g. disease vs no disease). This is NOT the same as only minimizing false positives – where this would involve finding out where on the ROC curve the false positive rate is zero (i.e. for this you would look at the ROC curve to work out what range of pass marks would yield a zero false positive rate (100% specificity) - but you’d also
probably get a low sensitivity at that point. This would be where the curve was touching the left hand side of the ROC curve graph. Statement 2 is correct – if the area under the ROC curve is .5, then this means that for any given pass mark the correct hit rate will match the false
positive rate.
Statement 1: A receiver operating characteristic curve can be used to help determine the pass mark of a diagnostic test.
Statement 2: To determine the pass mark for a diagnostic test that maximizes discrimination between people with and without a disease, you work out the point on a ROC curve where the value of the test sensitivity MINUS the test specificity is the highest.
(a) Both statements are true.
(b) Statement 1 true; Statement 2 false.
(c) Statement 1 false; Statement 2 true.
(d) Both statements are false.
The answer was b. See Lecture 12. Statement 1 is true. We can use a ROC curve to help us choose the pass mark of a diagnostic test. Statement 2 is false. To determine the pass mark for a diagnostic test that maximizes discrimination between people with and without a disease, you work out the point on a ROC curve where the value of the test sensitivity PLUS (NOT MINUS) the test specificity is the highest (i.e. the SUM of sensitivity and specificity).
A man is given a test for dementia. 1% of men in his age group have dementia. If he has dementia then the test will give him a positive result
90% of the time. If he does not have dementia then the test will give him a negative result 95% of the time. What is the chance the man has
dementia if he gets a positive test result?
(a) 18%
(b) 90%
(c) 9%
(d) 15%
The answer was d. See Lecture 12. Use the 2x2 table technique to calculate the answer to this question. From the question, sensitivity is 90%, specificity is 95%, and the base rate is 1%. Use those figures to populate the 2x2 table and you’ll find that only 15% of individuals with a positive test have schizophrenia. Don’t worry if your result isn’t exactly 15% - this could be due to your calculator doing some rounding of numbers.
A man is given a test for schizophrenia. 1% of men in his age group have schizophrenia. If he has schizophrenia then the test will give him a positive result 85% of the time. If he does not have schizophrenia then the test will give him a negative result 90% of the time. What is the
percentage chance the man has schizophrenia if he gets a positive test result?
(a) 8%.
(b) 9%.
(c) 15%.
(d) 10%.
The answer was a. See Lecture 12. Use the 2x2 table technique to calculate the answer to this question. From the question, sensitivity is 85%, specificity is 90%, and the base rate is 1%. Use those figures to populate the 2x2 table and you’ll find that only 8% of individuals with a positive test have schizophrenia. Don’t worry if your result isn’t exactly 8% - this could be due to rounding strategies.
