P-value

Question 1

What is the p-value?

Answer 1

The p-value is the proportion of students who answered an item correctly, so a measure of item difficulty.

The p-value is a value between 0 and 1.

A p-value of 1 would indicate that all students answered the item correctly.

Question 2

What are the optimal, minimum acceptable, and maximum acceptable p-values for a question with 2 answer options?

Answer 2

Optimal: 0.75
Minimum acceptable: 0.61
Maximum acceptable: 0.9

Question 3

What are the optimal, minimum acceptable, and maximum acceptable p-values for a question with 3 answer options?

Answer 3

Optimal: 0.67
Minimum acceptable: 0.5
Maximum acceptable: 0.9

Question 4

What are the optimal, minimum acceptable, and maximum acceptable p-values for a question with 4 answer options?

Answer 4

Optimal: 0.62
Minimum acceptable: 0.44
Maximum acceptable: 0.9

Question 5

What are the optimal, minimum acceptable, and maximum acceptable p-values for an open question?

Answer 5

Optimal: 0.5
Minimum acceptable: 0.25
Maximum acceptable: 0.9

Question 6

What is the corrected p-value? How is it calculated?

Answer 6

Pcorr is the p-value corrected for guess chance.

Pcorr = P-(1-P)(K-1), where K = the number of alternatives.

For example, if a question with 2 answer options has a p-value of 0.96, the corrected p-value is 0.92:
0.96-(1-0.96)(2-1)

Question 7

How is the optimal p-value calculated?

Answer 7

For an open question, the optimal score is 50% of the available points.

For a closed question, the optimal p-value is between the maximum p-value (1) and the guess chance. It’s calculated with (max p-value + guess chance)/2.

Question 8

Case study
The p-value is lower than, or the same as, the guess chance. What do you recommend?

Answer 8

Check the answer key - it is probably incorrectly labelled.
If it’s a trick question (ie. all students chose one other answer option), take it out.

Question 9

Case study
The p-value is higher than the guess chance, but much lower than the optimal value. What do you recommend?

Answer 9

This indicates a very difficult question.

If the Rit is sufficiently positive, then fine, but don’t have too many questions like this.

If the Rit is negative, reconsider the content of the question, adjust the answer options, or take it out.

Question 10

Case study
The p-value is (close to) 1. What do you recommend?

Answer 10

This indicates an extremely easy question that does not discriminate. However, that in itself is not a reason to remove it. Take a look at the exam as a whole.

Question 11

How many candidates do you need for a reliable p-value?

Answer 11

At least 25, though the p-values have less value than if there’s a large group (>100).