Lecture 2: Item response theory

Question 1

Describe a typical measurement model

Answer 1

It described the relationship between a construct and other variables, measured by test Bp. The latent variable θ is measured by three items Xp1, Xp2 and Xp3, which each carry with them error Ep1-3.

Question 2

Distinguish between the four types of latent variable models by how they’re used

Answer 2

Item response theory: Continuous latent variable, categorical observed data

Factor Analysis: Continuous latent variable, Continuous observed data

Latent class analysis: Categorical latent variable, categorical observed data

Latent Profile analysis: Categorical latent variable, continuous observed data

Question 3

What could scoring on a unidimensional IRT model look like?

Answer 3

The scoring is could be:
correct (0); false (1)
yes (1); no (0)
agree (1); disagree (0)

i.e unidimensional categorical data

Question 4

Describe a unidimensional IRT model in regards to its function, expected value and when it is suitable.

Answer 4

The measurement model is the expectation of the item score as a function of the latent variable:
E(Xpi | θp) = P(Xpi = 1|θp)

We know that Xpi is categorical and θp is continuous. By definition the expected value of a dichotomous variable (variable that only has 0 and 1 in it) is the same as the probability of scoring a 1 on that variable.

Therefore the focus is finding the probability of answering 1 to an item as a function of the latent trait. That probability as a function of the latent trait follows an S shaped curve (see doc); if you are high on the latent variable (e.g verbal ability) then you will have a score close to 1 (representing the ‘correct’ answer). Since it is a unidimensional model, it assumes that there is only one variable (verbal ability being measured).

Question 5

Is it common to model unidimensional IRT models?

Answer 5

Yes, it is uncommon to model multidimensional IRT models because IRT is typically used for very strict tests that need to measure one thing, e.g academic tests, however they do exist. They will not be discussed in this lecture however, and will be more discussed in the FA model lectures.

Question 6

Aside from correct, incorrect exams, when else are IRT models useful

Answer 6

In measuring a trait/ disorder, e.g depression or an opinion e.g death penalty etc.

Question 7

Since IRT is based on an S-shaped relation between Xpi and θp, we need an S shaped function.

Name the two most popular options for this function.

Answer 7

Normal ogive function (cumulative normal):
𝑓 𝑥 = Φ(𝑥) = 1 / sqrt(2𝜋) 𝑥S−∞ 𝑒^(−1/2)(ℎ^2) 𝑑ℎ

Logistic function:
𝑓 (𝑥) = 𝑒^𝑥/ 1+𝑒^𝑥
𝑓 (𝑥)= 1 / 1+𝑒−𝑥

Don’t have to know models by heart

Question 8

What does the normal give function do? Why is it suitable for IRT?

Answer 8

𝑓 𝑥 = Φ(𝑥) = 1 / sqrt(2𝜋) 𝑥~−∞ 𝑒^(−1/2)(ℎ^2) 𝑑ℎ

It uses a cumulative normal distribution and gives you the probability of finding a score ‘x’ or smaller (e.g probability table; p score).

If you plot it; put x on the x axis and the result of the function on the y axis you get this s shaped curve. This is logical because it approaches 1. This makes it optimal for IRT.

Question 9

Explain the logistic function

Answer 9

𝑓 (𝑥) = 𝑒^𝑥/ 1+𝑒^𝑥
𝑓 (𝑥)= 1 / 1+𝑒^−𝑥

𝑒 denotes a number (around 2.72) and this is just an exponential function. 𝑒^𝑥 is denoted to as exp(x).

Question 10

When do you use
𝑓 (𝑥) = 𝑒^𝑥/ 1+𝑒^𝑥

and when do you use
𝑓 (𝑥)= 1 / 1+𝑒^−𝑥

Answer 10

You can use either of the functions, they both do the same thing. Sometimes it is denoted as the first, sometimes as the second.

Question 11

Traditionally which function was used and when was did the other function come into play?

Answer 11

Traditionally IRT models have been developed using a normal ogive function. Later the logistic function was used as an approximation as it was easier to compute with.

Question 12

How was the logistic function adapted to make it closer to the normal ogive function?

Answer 12

If D = around 1.71 then:
𝑓 (𝑥) = 𝑒^D𝑥/ 1+𝑒^D𝑥

is very close to the normal ogive, as it makes the function more steeper (see doc)

Question 13

Why is the logistic function sometimes preferred to the normal ogive function?

Answer 13

Because the Normal Ogive function is quite complicated and the logistic function is less complex with less variables. This makes it easier to compute and also easier to make derivatives of the model, which is used a lot in modelling.

Question 14

How useful is the addition of the D to the logistic function and how is it useful?

Answer 14

It is not very useful, it is usually just used by people who want to stick to the old ogive framework. While it doesn’t matter at all; your model will be exactly the same, it will be conceptionally the same, your results will be the same. (Explained just in case you see it in some papers)

Question 15

What was the first IRT model ever developed? What is it known as now?

Answer 15

The original ‘Rasch model’:
𝑃 (𝑋𝑝𝑖 =1 |𝜃𝑝) = 𝑒^ 𝜃𝑝−𝑏𝑖 / 1+𝑒𝜃𝑝−𝑏𝑖

Also known as the one-parameter model:
𝑃 (𝑋𝑝𝑖 =1 |𝜃𝑝) = 𝑒^ 𝑎(𝜃𝑝−𝑏𝑖) / 1+𝑒^𝑎(𝜃𝑝−𝑏𝑖)

Question 16

Explain the one parameter model

Answer 16

𝑃 (𝑋𝑝𝑖 =1 |𝜃𝑝) = 𝑒^ 𝑎(𝜃𝑝−𝑏𝑖) / 1+𝑒^𝑎(𝜃𝑝−𝑏𝑖)

𝑒^ 𝑎(𝜃𝑝−𝑏𝑖) / 1+𝑒^𝑎(𝜃𝑝−𝑏𝑖) is the logistic function

Item parameters:
b tunes the location of the s shaped curve on the latent trait dimension (x axis), i.e if someone is high on a latent variable and therefore has a high probability of answering certain questions ‘correctly’. The curve shifts to the left/ right if you change b. b therefore denotes item difficulty: if the curve is more to the right then it is a more difficult item as even those with a high level of the latent trait struggle to get a higher score.

a denotes the slope/ steepness of all the items. The Rasch model does not account for that.

Question 17

How is score of the latent trait denoted on these graphs?

Answer 17

The mean 𝜃 is 0 and the SD is 1

Question 18

How can you measure probability using these graphs

Answer 18

You can draw vertically up from the theta score and horizontally across: e.g for bi = 1 and a latent 𝜃 score of 1, the probability for answering the item correctly is 0.5. b is located where the probability of a correct answer is 0.5 for a given score of theta in a two-parameter model.

Question 19

What does it mean to say that a and bi are fixed effects while 𝜃 denotes random effect?

Answer 19

a and bi are fixed effects while 𝜃 denotes random effect. This means that the latent variable has a (mostly normal) distribution since it is a sample from the population. The a and b parameters that you estimate in your model but typically do not have a distribution since b is about items and items are considered fixed. ai does not typically differ across items; it is fixed

Question 20

What name is given to this S shaped curves that the normal ogive function and the logistic function are attempting to mimic?

Answer 20

Item Characteristic Curve (‘ICC’)

Question 21

What changes in the 2 parameter model?

Answer 21

An item specific discrimination parameter ‘ai’ is introduced. This means that steepness can be specified for different items:

𝑃 (𝑋𝑝𝑖 =1 |𝜃𝑝) = 𝑒^ 𝑎i(𝜃𝑝−𝑏𝑖) / 1+𝑒^𝑎i(𝜃𝑝−𝑏𝑖)

Question 22

How does changing the value for ai mean for the function and how can it be interpreted?

Answer 22

a higher value for ai (e.g 10) means a steeper slope, a smaller value for ai (e.g 0.1). A negative value for ai would have a backwards curve and would typically be for a contra indicative item. The standard value for a before was 1.

ai tunes how well an item can distinguish between people who differ on the latent variable. A large ai means that small differences in the 𝜃 score have a large effect on the probability of answering the item correctly at a specific level of the latent trait denoted by bi. A small ai value means that small differences in 𝜃 does not have much effect on the probability of scoring the item correctly and therefore is poor at discriminating people according to 𝜃, however spans across a range of 𝜃 values.

ai is therefore known as the discrimination parameter.

Question 23

ai is the _______ parameter and bi is the ______ parameter

Answer 23

ai is the discrimination parameter and bi is the difficulty parameter

Question 24

Why does it make sense that a contra-indicative item would have a negative ai?

Answer 24

Because the probability of a ‘yes’/ correct response gets lower as you are higher on the latent trait

Question 25

Does the one-parameter model have a discrimination parameter?

Answer 25

Yes, but it is the same for all items

Question 26

How do you calculate the probability of someone scoring incorrectly on an item?

Answer 26

if the probability of scoring correctly on a two-parameter model is:
𝑃 (𝑋𝑝𝑖 =1 |𝜃𝑝) = 𝑒^ 𝑎i(𝜃𝑝−𝑏𝑖) / 1+𝑒^𝑎i(𝜃𝑝−𝑏𝑖)

then the probability of being incorrect is 1- prob(correct) i.e:
1−𝑃 (𝑋𝑝𝑖 =1 | 𝜃𝑝 ) = 1 - (𝑒^ 𝑎𝑖(𝜃𝑝−𝑏𝑖) / 1 + 𝑒^𝑎𝑖(𝜃𝑝−𝑏𝑖))
= 1 + 𝑒^ 𝑎𝑖(𝜃𝑝−𝑏𝑖) / 1 + 𝑒^𝑎𝑖(𝜃𝑝−𝑏𝑖) - 𝑒^ 𝑎𝑖(𝜃𝑝−𝑏𝑖) / 1 + 𝑒^𝑎𝑖(𝜃𝑝−𝑏𝑖)
= 1 / 1 + 𝑒^𝑎𝑖(𝜃𝑝−𝑏𝑖)

this is similar to the other form of the logistic function without the -ai but NOT the same. The other form of the logistic model has a negative exponential

Question 27

What changes in the three parameter model? What function does this serve?

Answer 27

The probability of guessing correctly e.g in a multiple choice questionnaire you will score some correctly even if you have no knowledge on the topic. It can also represent a factor such as fraud or plagiarism which explains a high probability despite a low theta score. Therefore the guessing parameter, ci is included in the model:

𝑃 (𝑋𝑝𝑖 =1 |𝜃𝑝) = ci + (1 - ci) (𝑒^ 𝑎i(𝜃𝑝−𝑏𝑖) / 1+𝑒^𝑎i(𝜃𝑝−𝑏𝑖))

ai, bi and ci are fixed effects while 𝜃 is still random effect.

Question 28

What does this guessing parameter mean for the function? (2)

Answer 28

It adds a lower bound to the item. If the item discrimination has a negative value, then it adds an upper bound to the item.

It also means that b is not the point of the probability = 0.5 anymore since ci affects the lower bound and therefore where 0.5 is located.

Question 29

Does ci increase item difficulty

Answer 29

There is a slight increase in item difficulty with an increase in the value of ci. Not really worth worrying about

Question 30

What is meant by a graded response model?

Answer 30

Where each item is scored according to ordinal responses e.g likert scales 1-5 for applicability, agreement etc. Designed for ordered categories

Question 31

What model family are graded response models part of and why? Describe what changes in the formula

Answer 31

Part of the cumulative model family because instead of the probability of modelling 1 or 0 𝑃 (𝑋𝑝𝑖 =1 |𝜃𝑝) , you model the probability of x being equal to c or higher (𝑃 𝑋𝑝𝑖 ≥𝑐 | 𝜃𝑝). This is known as accumulative probability. This accumulative probability is then subjected to a two parameter model:

𝑃 (𝑋𝑝𝑖 ≥𝑐 | 𝜃𝑝) = 𝑒^ 𝑎i(𝜃𝑝−𝑏𝑖𝑐) / 1+𝑒^𝑎i(𝜃𝑝−𝑏𝑖𝑐)

Where bic is now a difficulty parameter for both the item and the category. There was c - 1 category parameters where c is the number of categories. So a 5 point Likert scale would have 4 category parameters. This makes sense when you consider a one dimensional model having 1 difficulty parameter despite there being 0 and 1.

Question 32

How is the graded response model calculated

Answer 32

The model can be used to calculate the probability of scoring a particular category or higher. The location of the curve changes as c changes (e.g bi3; 𝑃 (𝑋𝑝𝑖 ≥3 | 𝜃𝑝) ) and also the question the function is answering. The question would be what the probability of scoring 3 or higher on item i is. This is the same for bi2 - 4 but different to bi5, as there is no score higher than 5 on a 5 point scale and the question would become simply the probability of scoring 5.

The functions bi1 (prob scoring higher than 2) until bi5 (*probability of scoring a 5) form 4 S shaped curves progressively further along the axis representing theta. This is not particularly useful however, as we want to find the probability of scoring a particular score, not that score or higher.

Thus the category probabilities are calculated:
𝑃 (𝑋𝑝𝑖 = 𝑐 | 𝜃𝑝) = 𝑃 (𝑋𝑝𝑖 ≥𝑐 | 𝜃𝑝) - 𝑃 (𝑋𝑝𝑖 ≥𝑐 + 1 | 𝜃𝑝)
e.g
𝑃 (𝑋𝑝𝑖 = 3 | 𝜃𝑝) = 𝑃 (𝑋𝑝𝑖 ≥3 | 𝜃𝑝) - 𝑃 (𝑋𝑝𝑖 ≥4 | 𝜃𝑝)
(because 𝑃(3,4,5)– 𝑃(4,5) = 𝑃(3))

Question 33

Describe the graphed functions of the graded response model

Answer 33

𝑃 (𝑋𝑝𝑖 = 1 | 𝜃𝑝) forms an S shaped curve towards negative values, 𝑃 (𝑋𝑝𝑖 = 5 | 𝜃𝑝) forms an S shaped curve toward higher positive values. Between these 𝑃 (𝑋𝑝𝑖 = 2 | 𝜃𝑝), 𝑃 (𝑋𝑝𝑖 = 3 | 𝜃𝑝) and 𝑃 (𝑋𝑝𝑖 = 4 | 𝜃𝑝) form normal distributions. 𝑃 (𝑋𝑝𝑖 = 5 | 𝜃𝑝) and 𝑃 (𝑋𝑝𝑖 = 1 | 𝜃𝑝) act as upper and lower bounds for your scores respectively, If they were normally distributed then it would be predicted that people very low or high on theta would reply with nothing.

Question 34

Name two more models belonging to other model families and the family it belongs to

Answer 34

Partial Credit Model: Adjacent category model family

Nominal Response Model: Baseline category model family

Question 35

What is characteristic of a partial credit model?

Answer 35

Its a model for ordered categories e.g subjects can obtain 0-3 points for a given question; like open questions in exams

Question 36

How does the partial credit model differ to that of the graded response model in regards to its parameters?

Answer 36

The curves produced by the model look quite the same however the interpretation of the parameters is slightly different. Since it is an adjacent category model, the b parameters are the points at which the curves of each point overlap. This still reflects item difficulty however as it still affects the placement of the curves on the x axis (higher, more to the right, more difficult)

Question 37

What is characteristic of a Nominal Response Model?

Answer 37

Baseline category model family: Model for unordered categories. E.g an arithmetic test with multiple choice

Question 38

Describe the nominal response model in terms of how it works and the subsequent function

Answer 38

Example question:
Which of these models does not have a bic parameter?
1. Three parameter model
2. Nominal response model
3. Two-parameter model
4. Graded response model

All the answers except 2 can be scored 0 yet the NRM models each of these answers. The graphs look the same as the previous two models with the multiple curves however there is no numeric order to these graphs. The s shaped curves may be x = 4 and x = 2 and those in between may be x = 1 and x = 3. E.g in the above example, 1 may be the s shaped curve on the left because it is the parameter in which bic is introduced and only those with very little knowledge on the subject would select this.

Question 39

There are a lot of models so there have been efforts made to reduce them. All models discussed, except ________ are _________ models

Answer 39

All models discussed, except the three parameter model are generalised linear models

Question 40

What is meant by a generalised linear model?

Answer 40

You can transform the probability of answering a 1 (or correct response) to get a linear function, For some function 𝑓(.), 𝑓 (𝑃 (𝑋𝑝𝑖 =1 | 𝜃𝑝 )) is a linear function.

E.g for the one and two parameter models:
Logit function: f(x) = logit(x) = log(x/1-x)

one parameter:
logit(P(Xpi = 1| 𝜃𝑝)) = … = log(𝑒^𝑎(𝜃𝑝−𝑏𝑖)) = a(𝜃𝑝−𝑏𝑖)

two parameter:
logit(P(Xpi = 1| 𝜃𝑝)) = … = log(𝑒^𝑎i(𝜃𝑝−𝑏𝑖)) = ai(𝜃𝑝−𝑏𝑖)

Question 41

When is the probit function used?

Answer 41

Something similar can be shown using the probit function, however the cumulative normal distribution is used. There the probit function is used for cumulative normal IRT models with the cumulative normal ogive functions §

Question 42

Name the link function and the form produced for each model

Answer 42

One-parameter model: Logit / Probit, 𝑎(𝜃𝑝 −𝑏𝑖)
Two-parameter model: Logit / Probit, 𝑎𝑖(𝜃𝑝 −𝑏𝑖)
Three-parameter model - -
Graded response model: Cumulative Logit / Probit, 𝑎𝑖(𝜃𝑝 −𝑏𝑖𝑐)
Partial credit model: Adjacent Logit / Probit, 𝑎𝑖(𝜃𝑝 −𝑏𝑖𝑐)
Nominal response model: Logit / Probit, 𝑎𝑖𝑐(𝜃𝑝 −𝑏𝑖𝑐)

Question 43

What is the equivalent of reliability (CTT) in IRT? How is this different?

Answer 43

Item information: The amount of information that an item provides for a given 𝜃𝑝. It is used for similar purposes as reliability however the important difference is that it gives you, loosely speaking, the reliability at a specific point of theta. In CTT you got the reliability of the test but it was mixed up with person properties and item properties etc. Here you look at an item rather than the whole test.

Question 44

What is the general item information formula for one and two parameter models?

Answer 44

𝐼𝑖(𝜃𝑝) = 𝑎𝑖^2𝑃𝑖(𝜃𝑝)𝑄𝑖(𝜃𝑝)

𝑃𝑖 (𝜃𝑝) = 𝑒^ 𝑎i(𝜃𝑝−𝑏𝑖) / 1 + 𝑒^ 𝑎i(𝜃𝑝−𝑏𝑖)
𝑄𝑖 (𝜃𝑝) = 1 / 1 + 𝑒^ 𝑎i(𝜃𝑝−𝑏𝑖)

where 𝑄𝑖 (𝜃𝑝) is the probability of an incorrect response, and 𝑎𝑖 =𝑎 for the one parameter model.

Question 45

What is the general item information formula for three parameter models?

Answer 45

𝐼𝑖(𝜃𝑝) = (𝑎𝑖^2) (𝑄𝑖 (𝜃𝑝) / 𝑃𝑖 (𝜃𝑝)) ({𝑃𝑖 (𝜃𝑝) −𝑐𝑖}^2/ (1 - cj)^2

Question 46

Describe an important characteristic of the item difficulty equation of the two parameter model. Briefly describe a second characteristic.

Answer 46

The item gives the most information at the item difficulty (bi, steepest point in s shaped curve; inflection point). This is intuitive because little differences in theta at this point make big changes to the probability of a correct answer compared to the two ends of the distribution.

The amount of information an item gives at the top of the item difficulty equals 1/4 ai^2. This is important because it means items with a large discrimination will give more information about the latent trait.

Question 47

What is different about the item information of three parameter models?

Answer 47

Maximim information is not at an items difficulty due to the lower bound changing the curve a bit. At the point b (difficulty parameter) you have a probability of ci + 1/2(1 - ci). In the item information function the maximum information is just above the item difficulty.

therefore the maximimum information of a three parameter model is b + … (don’t need to know exactly)

Question 48

How do the item information function change for contraindicative items

Answer 48

It doesn’t: the normal distributions look the same flipped and the negative values are squared so it doesn’t matter

Question 49

What are the item information values used to calculate? How is this achieved?

Answer 49

They are used to calculate the test information: the amount of information that a test provides for a given value of theta. The general formula is:
𝐼 𝜃𝑝 = E|n,𝑖=1| 𝐼𝑖 (𝜃)
where n is the number of items. ie just summing all the item information values

Question 50

What is meant by target information?

Answer 50

Desired information specified by the researcher. E.g wanting an intelligence test which covers the whole range of the latent variable would require specifying a target information function (looking at what shape you want your test information function to form), and fitting item information functions so that they make that shape (see docs). A target information function can also serve to have cut off points so that people at a particular value of theta are selected.

Question 51

How do you get item information from multiple category models?

Answer 51

Each response category gives information:
𝐼𝑖c(𝜃𝑝) = 𝑎𝑖^2𝑃𝑖c(𝜃𝑝)𝑄𝑖c(𝜃𝑝)

where 𝑃𝑖𝑐 𝜃𝑝 is the probability to respond in category 𝑐
and 𝑄𝑖𝑐 𝜃𝑝 is the probability to not respond in category 𝑐

Item information: 𝐼𝑖 𝜃𝑝 =E|𝐶, 𝑖=1| 𝐼𝑖𝑐 (𝜃)