Psychometrics - Test Items

Question 1

Where does the choice of format come from?

Answer 1

Objectives and purposes of the test

Question 2

Eight tips for item writing

Answer 2

1. Define clearly what you want to measure
2. Generate an item pool
3. Avoid long items
4. Keep the reading difficulty appropriate
5. Use clear and concise wording
6. Mix positively and negatively worded items
7. Cultural neutrality
8. Make the content relevant to the purpose

Question 3

Five categories of item format

Answer 3

1. Dichotomous
2. Polytomous
3. Likert
4. Category
5. Checklists + Q-sorts

Question 4

Examples of dichotomous format

Answer 4

True-False questions and Yes-No questions

Question 5

Pros and cons of Dichotomous format

Answer 5

+ Easy to administer and score
+ Participants can’t opt for neutral
- Less reliable, due to less range of scores
- Encourages memorization in test setting
- Doesn’t account for the fact that the truth is often in shades of grey, not black and white

Question 6

Example of polytomous format

Answer 6

MCQs

Question 7

Tips for writing distractors

Answer 7

Minimum of three distractors (so 4 options in total), less is an issue due to too few options and more is an issue due to the difficulty of writing viable options. They need to be as plausible as the correct answer and avoid cute distractors.

Question 8

Pros of polytomous format

Answer 8

+ Easy to administer and score
+ Requires absolute judgement
+ More reliable than dichotomous bc less change of guessing correctly

Question 9

Why do we do correction for guessing?

Answer 9

With dichotomous and polytomous formats, it’s easy for people to guess the right answer, which doesn’t indicate good performance, just luck.

Question 10

Formula for correction for guessing

Answer 10

Right-(Wrong/(number of alternatives - 1))

Question 11

Debate around the neutral option in Likert format

Answer 11

It allows for people to opt for an indecisive option instead of working on the answer.

Question 12

When do we use Likert?

Answer 12

When measuring the indicator of the degree of agreement, used in things like attitude or personality tests.

Question 13

Problems with category format

Answer 13

• Tendency to spread responses across all categories
• Susceptible to the context
• Element of randomness

Question 14

When do we use the category format?

Answer 14

• When people are highly involved with a subject

- When wanting to measure the amount of something

Question 15

Checklists vs Q-sorts

Answer 15

Checklist: you check the things on a list that apply to you

Q-sort: Places statements into piles the best describe them

Question 16

5 steps of item analysis

Answer 16

1. Item difficulty
2. Item discriminability
3. Item characteristic curves
4. Item response theory
5. Criterion referenced tests

Question 17

What is item difficulty?

Answer 17

Proportion of people who get the particular item right, so the higher the value, the easier the item.

Question 18

What is the optimum difficulty level?

Answer 18

Between 0.3 and 0.7

Question 19

How do we calculate optimum difficulty?

Answer 19

Halfway between 100% getting the item right and the level of success estimated by guessing, added to chance.

Question 20

Example of OLD equation in a 4 option MCQ

Answer 20

4 options = 0.25 chance of guessing right

1. (1-0.25)/2 = 0.375
2. 0.25 + 0.375 = 0.625

0.625 is our ODL

Question 21

When do we need to make exceptions about the ODL

Answer 21

More difficult: selection processes
Easier: special education
Others: boosting morale etc.

Question 22

What does item discriminability tell us?

Answer 22

Have those who have done well on the particular item done well on the test?

Question 23

What does the extreme groups method do?

Answer 23

Proportion of students in the upper extreme who got it right - the proportion of students in the lower extreme who got it right

Question 24

What does the point-biserial method do?

Answer 24

It gives us the item-total correlation; if there is a correlation between how well people did on the item and how well they did on the test as a whole

Question 25

Formula for point-biserial method

Answer 25

Correlation = ((Yn - Y)/Sy)(SQRT(Px/(1-Px))

Yn = mean tests score for those who got the item right 
Y = total mean 
Sy = stddev of all test takers 
Px = proportion of test takers who got the item correct

Question 26

What can we do with the point-biserial correlation?

Answer 26

Use it to weed out bad questions after the pilot study; include items with higher correlation and exclude the lower.

Question 27

What is plotted on the axes of the item characteristic curve?

Answer 27

X - total test score

Y - proportion getting the item correct

Question 28

How does item response theory work?

Answer 28

A computer generates items for the test taker depending on their performance on the previous items, with the outcome being defined by the level of difficulty of the items answered correctly.

Question 29

Pros of the item response theory

Answer 29

+ Tests based on IRT are great for computer administration
+ Quicker
+ Morale is maintained
+ Reduces cheating

Question 30

Three kinds of measurement precision

Answer 30

1. Peaked conventional: best for measuring the middle bracket, where average scores sit, it’s not good at measuring top and bottom achievers
2. Rectangular conventional: equal number of items assessing all ability levels, but relatively low precision across the board
3. Adaptive: test focuses on the range that challenges each individual test taker, making for overall high precision

Question 31

What is a criterion-referenced test designed to do?

Answer 31

Compare test performance with some objectively define criterion, tests developed based on learning outcomes

Question 32

How does one evaluate a criterion reference test?

Answer 32

Assess two groups, one given the learning unit and the other not. Collect their scores and put them on a graph.

Question 33

Limitations of the criterion referenced test

Answer 33

• Tells you what you got wrong, but not why
• Emphasis on the ranking of students, not identifying what gaps in their knowledge exist
• It increases the risk of “teaching to the test”