7 Test Theory Flashcards
In Classical Test Theory, what is the definition of item difficulty (p-correct)?
The proportion of people passing or ‘endorsing’ the item. The easier the item is to endorse, the higher the p value.
CTT: What is the p-correct in the following example?
I had two apples and Mary gave me three more apples, how many apples would I now have?
A) one apple n=30
B) five apples n=150
C) about 1kg of apples n=10
D) too many apples n=10
150/200 = .75 p = .75
In what way is item difficulty sample dependentin CTT?
Because the difficulty of an item depends on who is trying to solve it – e.g., 3-yr-olds vs. undergrads.
What is the ‘best’ item difficulty level in CTT? And why?
.5 –half the people get it correct
Why? This provides the maximal variation in test scores, and the best possibility of discrimination between participants.
What is wrong, from an item difficulty perspective, with items that everyone gets right or wrong?
Such items do not discriminate between individuals.
In CCT, p-correct is a feature of the ________, NOT an _______ characteristic of the item
In CTT, p-correct is a feature of the sample, NOT an absolute characteristic of the item.
In CTT, what is the Extreme Groups Approach to item discrimination?
Split the scores into two groups (e.g., top/bottom 25%)
D = P-upper – P-lower
Maximum discrimination is 1.
What is item-total correlation and what does it tell us?
The correlation between score of one item and the total score of all OTHER items.
It tells us the extent to which the single item is measuring the same thing as the total test.
What are inter-item correlations for?
They tell us whether items that should be measuring the same thing are related to each other.
What are four problems with classical test theory?
- Reliabilities and validities are derived from the total score
- Item selection is not fundamental
- Statistics are sample-dependent
- There is, in reality, no conceptual link between difficulty and discrimination (but CTT assumes there is)
In Item Response Theory, what forms the basis of estimating the trait level?
The person’s response pattern to a particular set of items. e.g. what level of extraversion is most likely to have led to this pattern of responses.
How does IRT scale the difficulty of an item?
As how much ability you need to get the item correct.
What do probability curves in IRT model?
The probability of getting an item correct given ability (N of items correct).
Are probability curves in IRT linear? Do they cross?
No, they are curvilinear. No they don’t cross.
What does the IRT model describe?
How changes in level of a trait relate to changes in probability of a certain response.
In IRT, what is the guessing parameter?
Probability of answering correctly given very low ability.
In IRT, what is the difficulty parameter?
Difficulty parameter is the amount of the attribute you need to have for a 50% chance of scoring 1 on the item (unlike the atheoretical view of CTT which is easy if lots pass it, hard if few do).
What is the slope in IRT?
Discrimination
What is the Rasch model of item response?
Item response is a function of the difference between the trait-level of the person and the difficulty of the item: θs - βi
e.g. how much ability is required to get that item right and how much of the ability you have
In the Rasch model, what are odds?
Odds = ratio of success to failures.
Odds of 4/1 means that…
Odds of 4/1 means that out of every 5
attempts, there are 4 success and 1 failure. Or, probability of success / probability of failure (.80/.20)
What is the dependent variable in the Rasch model?
Dependent Variable is the probability of correct response (i.e., a pass).
What does this formula mean? P(Xis =1|θs,βi)
Probability of the response (X) to item i by subject S is correct, given trait-level θ of Subject S and difficulty β of item i.
How is difficulty defined in the Rasch model?
Difficulty is the amount of ability needed to get .5 correct.
What is the principle of conjoint scaling in the Rasch model?
Trait-level and item difficulty are on the same scale.
In the Rasch model, when ability and difficulty match, what is your chance of a correct answer?
When ability and difficulty match,
you have .5 chance of correct answer.
