Test Development Flashcards
Item Analysis
Used to determine which items should stay in the test and which should be discarded
At a minimum, items should: • Correlate with each other • Correlate with the test total score • Discriminate between people who differ on the ability/skill/trait
At a minimum, items should: • C_________ with each other • C_________ with the test total score • D__________ between people who differ on the ability/skill/trait
Classic Test Theory (CTT) vs. Item Response Theory (I RT)
Two theories for test development: C______T_______T I_______R_______T
Classic Test Theory (CTT
Key principle is that item (test) score = true score + error (where error is any part of the score that has nothing to do with the test taker’s ability/level of the construct) _______-based item analysis focuses on the ability of items (tests) to accurately predict ‘true’ levels of the construct
Item Response Theory (IRT)
• Newer approach to item analysis • I RT-based item analysis focuses on using modelling to demonstrate how differing levels of a construct result in different responses to items • Works best for unidimensional tests and where there are correct/incorrect answers
The Item difficulty index - CCT
The I__________ d__________ i____________ Calculated on the proportion of the total number of test takers who got the item correct. Ranges from 0 (no one got it right) to 1 (everyone got it right).
The Item difficulty index - IRT
Probability of a test taker answering the item correctly. Can be used to target sub-groups.
Item discrimination Index
I___________ d __________ i indicates how adequately an item separates or discriminants between high and low scores on a test. Discriminates between high and low scorers. • Calculated by subtraction of% of test takers in lower group from% of test takers in the higher group • Positively discriminating item (positive number) means more high scorers got the item correct - item should be kept • Negatively discriminating item (negative number) means more low scorers got the item correct - item should be deleted or revised • Non-discriminating item (o or close to o) means that the item does not discriminate between high and low scorers- item should be deleted
IRT Item-Characteristic Curves (or Trace Lines) • Steeper slopes= greater item discrimination • Positive slope= more high scorers got it right/endorsed the item Negative slope = more low scorers got it right/endorsed the item
Graphical representation of item difficulty/endorsement and discrimination (IRT) • Steeper slopes= ________ item discrimination • Positive slope= more __________ scorers got it right/endorsed the item Negative slope = more _______ scorers got it right/endorsed the item
a) Rate of rise of the function (i.e. steepness of the curve)
• Indicates item discrimination
b) Position of the function on the horizontal (X) axis
• Indicates item difficulty
c) Position of the function in relation to the vertical (Y) axis
• Indicates guessing level
Rasch Model is 1PL
Shows relationship between the probability of a specific response (e.g.,
correct/incorrect) to a person’s level of a latent trait (construct) and die difficulty of the item
Item Reliability (CCT)
- Inter-item and item-total correlations
- Analysis of the extent to which items correlate with each other (inter-item) and with the scale total score (item-total)
- All correlations should be positive with appropriate strength - Inter-item: r > .30; Item-total: r > .70
Cronbach’s a
relatively poor reliability (e.g., a= .40 - .60) - may be because the test is measuring more than one clear construct
very high a coefficients (e.g., .95 and higher) - may indicate that there is some redundancy in your items (i.e., some items are too similar to each other)
