Exam 2 Flashcards
Designed to measure the typical behavior and characteristics of examinees
Typical response test
Item in which all scores agree on the score for the item
Objective item
Item in which disagreement may exist between scorers
Subjective item
Example of subjective items
Short answer, essays
Sample of objective items
Multiple-choice true false matching
Type of test that measures knowledge and skills in an area in which instruction has been provided
Achievement test
A type of test that measures cognitive abilities and skills that are accumulated as a result of overall life experience
Aptitude test
Type of test in which performance reflects differences in the speed of performance
Speed test
Type of test in which performance reflects the difficulty of the items the examinee is able to answer correctly
Power test
Type of personality test that uses items that are not influenced by the subjective judgment of the person scoring the test
Objective personality test
Type of test that involves the presentation of ambiguous material that elicits an almost infinite range of responses
Projective personality test
Types of maximum performance tests
Achievement/aptitude
Objective/subjective
Speed/power
Types of typical response test
Objective/projective
Item requires examinee to select a response from multiple alternatives
Selected response
Item requires examinee to create or construct a response
Constructed response
Strengths of selected response
You can typically include a relatively large number of selected-response items in your test
They can be scored in an efficient, objective, and reliable manner
They are flexible and can be used to assess a wide range of abilities
They can reduce the influence of certain construct-irrelevant factors
Limitations associated with the use of selected response items
They are relatively difficult to write
They are not able to assess all abilities
They are subject to random guessing
Strengths of constructed response items
Compared to selected response items they are easier to write
they are well suited for assessing higher order cognitive abilities and complex task performance
They eliminate random guessing
Limitations of constructed response items
They require more time then selected response to answer and therefore cannot include as many items in a test
They are more difficult to score in a reliable manner
They are vulnerable to feigning
They are vulnerable to the influence of construct-irrelevant factors
12 Item writing guidelines
Provide clear directions Use clear east to understand language Develop items that can be scored in a decisive manner Avoid cues to answers Arrange items in a systematic manner Try to contain similar items on the same page Tailor items to target population Minimize construct-irrelevant factors Avoid using exact phrasing from textbook Avoid biased language Make things easy to read Determine how many items to include
5 Types of items included in maximum performance tests
Multiple choice True/False Matching Essay Short answer
Type of multiple choice test when there is only one right answer
Correct answer test
Type of multiple choice test when there is more than one correct answer and the objective is to identify the best answer
Best answer format
Incorrect answers
distractors
What is the most popular type of selected response test
Multiple Choice
11 Guidelines for developing multiple choice items
Make item as clear as possible
Stem should contain all info necessary to understand problem
3-5 alternatives
Keep alternatives brief
Avoid negatively stated stems
Make sure there is only one correct choice
The alternatives should be grammatically correct with the stem
Distractors should appear plausible
Don’t make the same alternative the correct answer every time
Minimize the use of “non of the above”. Avoid using “all of the above”
Limit the use of “always” or “never” in the alternatives
What is the second most popular selected-response format
True-false
What is another name for true/false
Binary items
What are the 4 guidelines for developing True/false items
Avoid double barreled items
Avoid specific determiners/qualifiers that cue the answer
Make true and false statements the same length
Include equal number of true and false statements
Items that contain two columns of words and phrases.
Matching items
In matching items it is the column on the left for which the examinee seeks a match
premises
In matching it is the column on the right that the examinee uses to find a match
responses
6 Guidelines for matching items
Limit matching items to homogenous material
In the directions, indicate the basis for matching premises with responses
Include more responses than premises
Indicate if responses could be used more than once or not at all
Keep list brief
Ensure that responses are brief and follow a logical order
Poses a question or problem for the examinee to respond to in an open-ended written format
Essay item
Structured items that specify the form and scope of a response
Restricted response
Item in which there is no limit or form on the scope of response
Extended response
What is the benefit of restricted response items
Can be answered in a timely manner and are easier to score
What is the benefit of extended response items
Gives examinees more freedom in constructing a response
4 essay item guidelines
Clearly specify the assessment task
Use more restricted-response items in place of smaller number of extended response items
Develop and use a scoring rubric
Limit essay items to objectives that cannot be measured with selected response
Items that require the examinee to supply a word, phrase, number, or symbol in response to a direct question
Short answer item
Items written as incomplete sentences
Completion items
7 Guideline for short answer items
The response should be as short as possible
Make sure there is only one correct response
Direct-question format is preferable to the incomplete-sentence format
When using the incomplete-sentence format, it is best to have only one blank space, generally near the end of a sentence
Make sure that the blanks provide adequate space for response
For questions requiring quantitive answers, indicate the degree of precision expected
Create a scoring rubric and use it
Typical response item formats
True/false
Rating scale
Likert scale
Scale that focuses on frequency
Rating scale
Scale that focuses on degree of agreement
Likert scale
The percentage of test takers who correctly answer the item
Item difficulty index
What letter represents item difficulty index
P
What does P=
number of people who got the item correct over the number of total test takers
What is the range of the difficulty index
0.0-1.0
The percentage of examinees that responded to an item in a given manner
percent endorsement
Do easier items have higher or lower values
higher
To maximize variability and reliability items should be at what item difficulty index
.40-.60
On a mastery test what is the typical item difficulty index
.90
Why do mastery tests have such high difficulty indexes
because they are usually pass/fail and they indicate the extreme upper level of knowledge
Indicator of how well an item can differentiate among test takers who differ on the construct measured by the test
Item discrimination
What are the indexes of item discrimination
Discrimination index
Item-total correlation
What will the item discrimination likely indicate on a speed test
Where the item was placed in the exam because only the fastest examinees can reach the items at the end of the test
Index of the difference in performance between two groups
Discrimination Index
How do you calculate the discrimination index
Take the difficulty index of the item for each group, then subtract one group from the other
What letter represents the discrimination index
D
What is considered an excellent discrimination index
.40 or larger
What is considered a good discrimination index
.30-.39
What is considered a fair discrimination index
.11-.29
What is considered a poor discrimination index
0.00-.10
What does a negative discrimination index indicate
That an item was miskeyed or has a major flaw
How do you calculate the discrimination index for mastery testing
test two groups. one group receives instruction, the other does not. The same formula is used to compare the two groups
Correlation of the performance on the whole test to one item
Item-total correlation
Total number of items answered correctly including the item being looked at
Unadjusted item-total correlation
Total number of items answered correctly, omitting the item being examined
Adjusted item-total correlation
How is item-total correlation calculated
with point-biserial correlation
How does the item-total correlation indicate if an item measures the same construct as the test
The larger the item-total correlation, the more evidence that an item measures the same construct as the test
Does item-total correlation always indicate that your test measure what it intends to measure
No, it just shows that the item and the test measure the same thing
Examines how many people in the top and bottom groups selected each option on a multiple choice exam
Distractor analysis
Responses to items on a test that are accounted for by latent traits
Item response theory
A graph with ability on the X-axis and the probability of a correct response on the y-axis
Item Characteristic curve (ICC)
One parameter IRT model: items differ only by one parameter, difficulty
Rasch model
What do the lines look like on the Rasch model
They have the same slope
Items differ in both difficulty and discrimination
Two parameter model