Study Guide 3: Test Development, Pilot Testing, and Item Analysis Flashcards
Acquiescence bias
tendency to agree with any ideas or behaviors presented. Ex. Labeling each statement as true. Solution – Balance items with T and F responses.
Cronbach’s Alpha-if-item-deleted
tells us the reliability estimate that we would obtain for the test if we drop each item from the test.
Balanced scale
having a balance of questions that require true or false answers to avoid acquiescence.
Corrected item-total correlation
This is the correlation between an item and the rest of the exam, without that item considered part of the exam. If the correlation is low for an item, this means the item isn’t really measuring the same thing the rest of the exam is trying to measure.
Cross-validation
A method of testing validity by using more than one sample of people from the same population.
Distracter
the items that are incorrect on a multiple choice question, though they are designed to appear correct to someone who doesn’t know the correct answer
Extreme (or moderate) response bias
the tendency to use or avoid extreme response options.
Inter-item correlation matrix
the matrix displays the correlation of each item with another item. each is a phi coefficient – a result of correlating two dichotomous variables. These create a matrix. Ideally, each item should be highly correlated with every other item to increase the test’s internal consistency.
Item bias
an item’s being easier for one group than for another. Item characteristic curves are useful for determining this.
Item characteristic curve
– the line that results when we graph the probability of answering an item correctly with level of ability on the construct being measured. It provides a picture of both the item’s difficulty and discrimination. Difficulty is determined by the location of the point at which the curve indicates a probability of .5 (a 50-50 chance) for answering correctly. The higher the ability level associated with this point the more difficult the question.
Item difficulty
the percentage of test takers who respond correctly. Calculated as a p value, by dividing the number of persons who answered correctly by the total number of persons who responded to the question. Items with p values of .5 yeild distributions of test scores with the most variations. .2 (too difficult), or .8-1.0 (too easy).
Item discrimination
the degree to which an item might affect a test’s internal consistency- but the degree to which an item differentiates people who score high on the total test from those who score low on the total test. Prefer high values
Item discrimination index
– a statistic that compares the performance of the upper group who had very high test scores with the performance of the lower group who had low test scores D= U(Number in upper group who responded correctly/Total number in upper group)-L(Number in lower group who responded correctly/Total number in lower group) – upper and lower groups may be determined by 25-35th percentiles. percentiles.
Item-total correlation
a way of operationalizing an item’s discrimination. Comput the total score on a test, and then compute the correlation between an item with this total test score. It reflects the degree to which differences among persons’ responses to the item are consistent differences in their total test scores.
Malingering
faking bad. When respondents might attempt to exaggerate their psychological problems.