Writing and evaluating test items

Question 1

What determines your choice of item format?

Answer 1

Choice of item format comes from objectives and purpose of the test.

Question 2

What are the six item writing guidelines? Plus two

Answer 2

1. Define clearly what you want to measure.
2. Generate an item pool.
3. Avoid long items
4. Keep the reading difficulty appropriate.
5. Use clear and concise wording/avoid double-barreled questions.
6. Consider mixing negatively and positively worded items.
7. Make sure your items are as culturally neutral as possible.
8. Make the content relative to the purpose.

Question 3

What is meant by “define clearly what you want to measure”?

Answer 3

• You can’t construct a test/questionnaire items if you don’t know what it is you are trying to measure.
• To clearly define what you want to measure, use substantive theory as a guide and try to make your items as specific as possible.

Question 4

What is meant by “generate an item pool”?

Answer 4

• In the beginning stages you want to create a larger sample pool of items. From there you test the items on a representative sample group to decide which ones to include in your final measure.
• In the beginning stages, you might want to write 3 or 4 items more for each one that you will include in your final test.

Question 5

What is meant by “avoid long items”

Answer 5

• Long items are tedious to read and respondents may not get what you are trying to ask properly as they skim over the question.
• These types of items are rarely good.

Question 6

What is meant by “keep the reading difficult appropriate”?

Answer 6

• Use appropriate vocabulary.
• Avoid using jargon that would not be known to the sample.
• Take your sample’s level of education into account.

Question 7

What is meant by “use clear and concise wording”

Answer 7

• Avoid double-barreled items (e.g. is this tool interesting and useful)
• Avoid double negatives (e.g. it is not unfortunate that…)
• Make sure your wording is grammatically correct.

Question 8

What is meant by “consider mixing positively and negatively worded items in the same test”?

Answer 8

• This can help to prevent response sets.

Question 9

What are the item writing guidelines for MCQ items?

Answer 9

• Vary the position of the correct answer
• All distractors should be plausible (don’t make them obviously wrong)

Question 10

What are the item writing guidelines for True/False questions?

Answer 10

• Both statements should be the same length.
• Have equal numbers of both.

Question 11

What are the 5 different types of item formats?

Answer 11

• Dichotomous format
• Polytomous format
• The Likert format
• The category format
• Checklists and Q-sorts.

Question 12

What is the dichotomous format?

Answer 12

• It offers two alternatives for each format.

Question 13

What are the advantages of the dichotomous format?

Answer 13

• It is easy to administer and score
• It requires absolute judgement (the test taker must declare one of the two alternatives)

Question 14

What are the disadvantages of the dichotomous format?

Answer 14

• It is less reliable (test-takers have a 50% chance of getting an item correct. There is also less of a range of scores when it comes to doing analyses).
• Encourages memorization which means that test takers can outperform their level of understanding.
• Often truths come in shades of grey and not black and white. So it is not always easy to set questions in this format.

Question 15

What is the polytomous format?

Answer 15

• This format offers more than two alternatives.

Question 16

What are distractors and how are they used?

Answer 16

• Distractors are incorrect alternatives
• Tests can retain their psychometric properties with as few as three distractors
• Must ensure that distractors are as clearly written and as plausible as the correct answer.
• Avoid using cute distractors.

Question 17

What do we know about the polytomous format in educational settings?

Answer 17

• 4 alternatives are commonly used in educational settings
• Psychometric theory suggests that more distractors make the item more reliable, however it is difficult to find many good distractors
• 3-4 good distractors seem to be ideal.

Question 18

What are the advantages of the polytomous format?

Answer 18

• It is easy to administer and score
• It requires absolute judgement.
• It is more reliable than the dichotomous format as there is a lower chance of guessing the correct answer.

Question 19

What are the disadvantages of the polytomous format?

Answer 19

• Poorly written distractors can adversely affect the quality of the test. If distractors are too easy then a poorly prepared test taker could still guess the correct answer; therefore lowering the validity and reliability of the test.
• There is still a chance that someone could correctly guess the answer, so we need to correct for guessing.

Question 20

How do we correct for guessing?

Answer 20

• By using the formula: R-(W/(n-1)). The number of right answers minus the number of wrong answers divided by the number of choices for each item minus 1.
• R is the number of correct answers
• W is the number of wrong answers
• N is the number of alternatives
• Omitted answers are excluded in this calculation as they provide neither credit nor penalty.

Question 21

What is the difference in correction for guessing for polytomous and dichotomous formats?

Answer 21

• The option for dichotomous questions has a bigger correction because it is much easier to guess.

Question 22

What is the likert format?

Answer 22

• It was named after Likert who first used it as an attitudinal scale.
• Test-takers must indicate their degree of agreement with a statement.
• It is popular in attitude or personality scales.
• 6 point scale (or even number of options) is used to avoid the neutral response.
• Negatively worded items must be reverse scored.

Question 23

What are some issues surrounding the likert format?

Answer 23

• It is difficult to determine how many anchors to use.
• Items need to be statements instead of questions.
• Wording of response options has to be consistent throughout the measure.

Question 24

What is the category format?

Answer 24

• It is similar to the Likert format, but uses more choices. A popular number of choices is 10.

Question 25

What are the problems with the category format?

Answer 25

• People have a tendency to spread their responses across all categories.
• It is susceptible to the groupings of things being rated (e.g. if we are rating attractiveness on a scale from 1 to 10, we are naturally inclined to compare our ratings to others we are rating.
• There is an element of randomness.

Question 26

How do we solve some of the issues for the category format?

Answer 26

• The issue of rating people according to context, or wanting to spread out your answers evenly can be solved by clearly defining the endpoints of the scale.

Question 27

When is the category format used?

Answer 27

• When people are highly involved with a subject. Thi is because they are more motivated to make a finer distinction. E.g. asking people in townships to rate service delivery.
• When you want to measure the amount of something. E.g. road rage experienced in a given situation. You would rate how much different scenarios make you upset.

Question 28

What is an approach related to the category format?

Answer 28

• The visual analogue scale
• They are often used with children
• A Visual Analog Scale (VAS) is a simple but valuable instrument that tries to measure a characteristic or attitude that is believed to range across a continuum of values and cannot easily be directly measured.
• It usually is a horizontal line, 100 mm in length, anchored by word descriptors at each end, for example, that represent the severity of symptoms from 0 “no symptoms” to 10 “very severe symptoms.” The patient marks on the line the point that the patient believes represents his or her perception of his or her current state.

Question 29

What is a checklist?

Answer 29

• These are common in personality measures.
• The test taker is presented with a list of adjectives and they check off the ones they think best describes them.

Question 30

What is a Q-sort?

Answer 30

• Items are placed into piles.
• The piles indicate the degree to which the test taker thinks a statement describes them or another person
• The category format is implicit here.
• Or: sort into agree, disagree, and neutral. Then maybe rank those piles according to how much you agree or disagree with them.
• The frequency of items placed in each of the categories usually looks like a bell curve.
  *note: this is usually done on a 1-9 scale.

Question 31

What are some things to look out for with person-based questionnaires?

Answer 31

Some respondents will complete a questionnaire in a certain way regardless of the content of the items:

Question 32

What is acquiescence?

Answer 32

• The tendency to agree with items regardless of their content.
• This can be reduced by including negatively worded items.

Question 33

What is social desirability?

Answer 33

• The tendency to respond to an item in a socially desirable manner.
• This can be reduced by excluding items which are clearly socially desirable or undesirable.

Question 34

What is indecisiveness?

Answer 34

• The tendency to us the don’t know or uncertain option.
• This is a common problem which can easily be eliminated by removing the neutral anchor.

Question 35

What is extreme response?

Answer 35

• The tendency to choose the extreme option regardless of direction.
• This can be reduced by the use of clear, unambiguous and specific items.