Writing and evaluating test items Flashcards
1
Q
What determines your choice of item format?
A
Choice of item format comes from objectives and purpose of the test.
2
Q
What are the six item writing guidelines? Plus two
A
- Define clearly what you want to measure.
- Generate an item pool.
- Avoid long items
- Keep the reading difficulty appropriate.
- Use clear and concise wording/avoid double-barreled questions.
- Consider mixing negatively and positively worded items.
- Make sure your items are as culturally neutral as possible.
- Make the content relative to the purpose.
3
Q
What is meant by “define clearly what you want to measure”?
A
- You can’t construct a test/questionnaire items if you don’t know what it is you are trying to measure.
- To clearly define what you want to measure, use substantive theory as a guide and try to make your items as specific as possible.
4
Q
What is meant by “generate an item pool”?
A
- In the beginning stages you want to create a larger sample pool of items. From there you test the items on a representative sample group to decide which ones to include in your final measure.
- In the beginning stages, you might want to write 3 or 4 items more for each one that you will include in your final test.
5
Q
What is meant by “avoid long items”
A
- Long items are tedious to read and respondents may not get what you are trying to ask properly as they skim over the question.
- These types of items are rarely good.
6
Q
What is meant by “keep the reading difficult appropriate”?
A
- Use appropriate vocabulary.
- Avoid using jargon that would not be known to the sample.
- Take your sample’s level of education into account.
7
Q
What is meant by “use clear and concise wording”
A
- Avoid double-barreled items (e.g. is this tool interesting and useful)
- Avoid double negatives (e.g. it is not unfortunate that…)
- Make sure your wording is grammatically correct.
8
Q
What is meant by “consider mixing positively and negatively worded items in the same test”?
A
- This can help to prevent response sets.
9
Q
What are the item writing guidelines for MCQ items?
A
- Vary the position of the correct answer
- All distractors should be plausible (don’t make them obviously wrong)
10
Q
What are the item writing guidelines for True/False questions?
A
- Both statements should be the same length.
- Have equal numbers of both.
11
Q
What are the 5 different types of item formats?
A
- Dichotomous format
- Polytomous format
- The Likert format
- The category format
- Checklists and Q-sorts.
12
Q
What is the dichotomous format?
A
- It offers two alternatives for each format.
13
Q
What are the advantages of the dichotomous format?
A
- It is easy to administer and score
- It requires absolute judgement (the test taker must declare one of the two alternatives)
14
Q
What are the disadvantages of the dichotomous format?
A
- It is less reliable (test-takers have a 50% chance of getting an item correct. There is also less of a range of scores when it comes to doing analyses).
- Encourages memorization which means that test takers can outperform their level of understanding.
- Often truths come in shades of grey and not black and white. So it is not always easy to set questions in this format.
15
Q
What is the polytomous format?
A
- This format offers more than two alternatives.
16
Q
What are distractors and how are they used?
A
- Distractors are incorrect alternatives
- Tests can retain their psychometric properties with as few as three distractors
- Must ensure that distractors are as clearly written and as plausible as the correct answer.
- Avoid using cute distractors.
17
Q
What do we know about the polytomous format in educational settings?
A
- 4 alternatives are commonly used in educational settings
- Psychometric theory suggests that more distractors make the item more reliable, however it is difficult to find many good distractors
- 3-4 good distractors seem to be ideal.
18
Q
What are the advantages of the polytomous format?
A
- It is easy to administer and score
- It requires absolute judgement.
- It is more reliable than the dichotomous format as there is a lower chance of guessing the correct answer.
19
Q
What are the disadvantages of the polytomous format?
A
- Poorly written distractors can adversely affect the quality of the test. If distractors are too easy then a poorly prepared test taker could still guess the correct answer; therefore lowering the validity and reliability of the test.
- There is still a chance that someone could correctly guess the answer, so we need to correct for guessing.
20
Q
How do we correct for guessing?
A
- By using the formula: R-(W/(n-1)). The number of right answers minus the number of wrong answers divided by the number of choices for each item minus 1.
- R is the number of correct answers
- W is the number of wrong answers
- N is the number of alternatives
- Omitted answers are excluded in this calculation as they provide neither credit nor penalty.
21
Q
What is the difference in correction for guessing for polytomous and dichotomous formats?
A
- The option for dichotomous questions has a bigger correction because it is much easier to guess.
22
Q
What is the likert format?
A
- It was named after Likert who first used it as an attitudinal scale.
- Test-takers must indicate their degree of agreement with a statement.
- It is popular in attitude or personality scales.
- 6 point scale (or even number of options) is used to avoid the neutral response.
- Negatively worded items must be reverse scored.
23
Q
What are some issues surrounding the likert format?
A
- It is difficult to determine how many anchors to use.
- Items need to be statements instead of questions.
- Wording of response options has to be consistent throughout the measure.
24
Q
What is the category format?
A
- It is similar to the Likert format, but uses more choices. A popular number of choices is 10.
25
What are the problems with the category format?
- People have a tendency to spread their responses across all categories.
- It is susceptible to the groupings of things being rated (e.g. if we are rating attractiveness on a scale from 1 to 10, we are naturally inclined to compare our ratings to others we are rating.
- There is an element of randomness.
26
How do we solve some of the issues for the category format?
- The issue of rating people according to context, or wanting to spread out your answers evenly can be solved by clearly defining the endpoints of the scale.
27
When is the category format used?
- When people are highly involved with a subject. Thi is because they are more motivated to make a finer distinction. E.g. asking people in townships to rate service delivery.
- When you want to measure the amount of something. E.g. road rage experienced in a given situation. You would rate how much different scenarios make you upset.
28
What is an approach related to the category format?
- The visual analogue scale
- They are often used with children
- A Visual Analog Scale (VAS) is a simple but valuable instrument that tries to measure a characteristic or attitude that is believed to range across a continuum of values and cannot easily be directly measured.
- It usually is a horizontal line, 100 mm in length, anchored by word descriptors at each end, for example, that represent the severity of symptoms from 0 “no symptoms” to 10 “very severe symptoms.” The patient marks on the line the point that the patient believes represents his or her perception of his or her current state.
29
What is a checklist?
- These are common in personality measures.
- The test taker is presented with a list of adjectives and they check off the ones they think best describes them.
30
What is a Q-sort?
- Items are placed into piles.
- The piles indicate the degree to which the test taker thinks a statement describes them or another person
- The category format is implicit here.
- Or: sort into agree, disagree, and neutral. Then maybe rank those piles according to how much you agree or disagree with them.
- The frequency of items placed in each of the categories usually looks like a bell curve.
*note: this is usually done on a 1-9 scale.
31
What are some things to look out for with person-based questionnaires?
Some respondents will complete a questionnaire in a certain way regardless of the content of the items:
32
What is acquiescence?
- The tendency to agree with items regardless of their content.
- This can be reduced by including negatively worded items.
33
What is social desirability?
- The tendency to respond to an item in a socially desirable manner.
- This can be reduced by excluding items which are clearly socially desirable or undesirable.
34
What is indecisiveness?
- The tendency to us the don't know or uncertain option.
- This is a common problem which can easily be eliminated by removing the neutral anchor.
35
What is extreme response?
- The tendency to choose the extreme option regardless of direction.
- This can be reduced by the use of clear, unambiguous and specific items.
