Chapter 6: Test Development

Question 1

6 steps of test construction

Answer 1

1. Define the test (what are we testing and why)
2. Select item format
3. Construct test items
4. Test the items (determine reliability and validity)
5. Revise the test
6. Publish the test

Question 2

Answer choice formats: selected vs. constructed items

Answer 2

Selected items: pick from a number of answers (multiple choice, true/false, matching)
Constructed items: generate your own answers (short answer, essay)

Question 3

Strengths of selected-response items

Answer 3

Can include more items (each question takes less time to answer)
Increased content sampling as well as reliability and validity
Reduction of construct-irrelevant factors
Scoring is efficient and reliable

Question 4

Weaknesses of selected-response items

Answer 4

Developing items is time consuming (easier to write constructed items)
Unable to assess all abilities
Subject to random guessing (make it look like examinee knows more than he/she actually does)

Question 5

Strengths of constructed-response items

Answer 5

Questions are relatively easy to write
Can assess higher-order cognitive abilities (have to show reasoning)
No random guessing

Question 6

Weaknesses of constructed-response items

Answer 6

Test can include relatively few items (takes longer to answer each one)
Difficult to score reliably (even with good rubric, still hard)
Subject to misinterpretation (examinee might misconstrue question)
Construct-irrelevant factors can sneak in (ex- bad handwriting makes answers hard to read)

Question 7

Which three things on a test should be clear?

Answer 7

Clear directions (examinee should know how to answer question)
Clear questions (questions should only ask 1 thing; answering should be able to be done in a decisive manner)
Clear print (should be easy to read)

Question 8

5 things that should not be included on a test

Answer 8

Cues to answers (ex- including answer in a different question)
Items that cross pages (increases likelihood of examinee error)
Construct-irrelevant factors
Exact phrasing from materials (encourages rote memorization over understanding of concept)
Biased language and content

Question 9

2 things to consider when placing items on a test

Answer 9

Item arrangement: placement should make sense

Number of items: if using a power test, should be able to complete questions in given time limit

Question 10

Who the items on the test should be tailored to

Answer 10

The target population (example: wouldn’t give a college level test to 4th graders)

Question 11

Multiple choice tests: factors to consider pertaining to item stem

Answer 11

Stem should clearly state question

Negatively stated stems should not be used

Question 12

Multiple choice tests: factors to consider pertaining to alternatives

Answer 12

Alternatives should be brief
3-5 alternatives should be included
Alternatives should be grammatically correct in the question
Alternatives should be plausible (otherwise, question becomes a dead giveaway)

Question 13

Multiple choice tests: factors to consider pertaining to questions themselves

Answer 13

Items should be clear and easy to read
Only 1 correct/best answer should be included
Placement of correct answer should be random (otherwise, examinees can detect pattern)
Minimize “none/all of the above” and “always/never” questions (becomes a dead giveaway)

Question 14

True/false tests: factors to consider

Answer 14

Include only 1 idea in each item
Avoid specific determiners (never, always) and qualifiers (usually)
Keep items about the same length
Approximately equal number of true and false items

Question 15

Matching tests: factors to consider pertaining to directions

Answer 15

State basis for matching

Indicate responses may be used once, more than once, or not at all

Question 16

Matching tests: factors to consider pertaining to item design

Answer 16

Include more responses than stems (make it possible to get only 1 wrong)
Keep lists relatively short
Keep responses brief
Consider order of responses

Question 17

What kind of material should be used in a matching test?

Answer 17

Homogenous material (all items should relate to a common theme)

Question 18

Essay tests: factors to consider

Answer 18

Clearly specify the task
Develop a comprehensive scoring rubric
Limit essay items to objectives that can’t be easily measured with selected-response items
Grade blindly

Question 19

Short-answer tests: factors to consider

Answer 19

Items should only require short answers
Only 1 correct response
Quantitative items: indicate level of precision desired
Use direct question as opposed to incomplete sentences

Question 20

Factors to consider if using incomplete sentence short-answer questions

Answer 20

Include only one blank space (becomes dead giveaway otherwise)
Add blanks near end of sentence
Don’t give clues with blank size
Give enough space to answer

Question 21

Short-answer tests: what should be created for each item?

Answer 21

Scoring rubric

Question 22

Typical response tests: factors to consider pertaining to items

Answer 22

Focus items on experiences (thoughts, feelings, behaviors)
Limit items to a single experience
Avoid items that will be answered universally the same
Avoid leading questions
Limit use of “never” and “always”

Question 23

Typical response tests: factors to consider pertaining to responses

Answer 23

Don’t always have high numbers represent the same thing
Odd or even number of options: neutral option requires odd number
Label options for Likert-type scales (rating from 0-7, etc.)
Consider structuring scale as an interview