lec 9 Flashcards
Source material checklist:
- The item cannot be answered without using the source material
(the source material should be essential for the item). - The source material is familiar but novel and credible.
- The source material is thought-stimulating about a topic.
- The source material is brief and at an appropriate reading level.
- The source material (e.g., book, website, or self-made) is cited
properly (i.e., respectful of copyright law) - Text-based source material has correct spelling and grammar
(unless errors are deliberate).
characteristics of quality source-based multiple choice items:
Test the knowledge and skills you desire, NOT the knowledge presented explicitly in the source
Do not include extraneous clues
Must be at the applying or above
taxonomical level
Follow same guidelines as for selected response item construction
strengths of source-based MC items:
An efficient way to
measure the
interpretation of
information in various
forms (e.g., written,
charts, graphs, maps,
pictures).
Can measure
complex learning
outcomes.
Introductory material
provides a common
basis for responding.
Scoring is easy,
objective, and
reliable.
limitations of source based multiple choice items:
Difficult to construct
quality items using
novel introductory
material.
Written material is
highly dependent on
reading skills.
Highly subject to
extraneous clues.
Ineffective in
measuring the ability to
originate, organize,
and express ideas.
source based multiple choice item is technically sound when it:
is free of verbal clues to the
answer
uses source material that is
familiar but new
uses source material that is
brief and at an appropriate
reading level
cites source material
cannot be answered without
the source material
is at or above the applied
taxonomy level
has correct spelling and
grammar
meets the technically sound
checklist for multiple choice
items
a source based multiple choice item is equitable when the item is:
free from racial, ethnic,
gender, sexual, religious or
other bias
free of irrelevant material
stated in appropriate and
clear language
free from pop cultural
references that would not
be familiar to all students
what should you avoid having within the same answer key? (Multiple choice)
having too many consecutive responses
(e.g five items that all have answer key B.)
Reasonable limit is three consecutive items with the same key.
Two myths about multiple choice items:
o The longest response option is most likely to be correct.
o If you don’t know the answer, pick “C”.
make sure that distractors and keyed responses are ordered
logically
some items should ___ pyramided
NOT BE
distractors use student:
errors and misconceptions
make sure that the alternatives are
homogenous and parallel in structure
alternatives should be arranged in a _______ order
logical order
o Numerical order, if alternatives are numeric
o Alphabetic, if alternatives are text-based
o By length of the alternatives
- Pyramid: Sorting from the shortest alternative to the longest
- Reverse-pyramid: Sorting from the longest alternative to the
shortest
all of the above and none of the above should be used
judiciously (i.e., not used only when it is the keyed response – > test wiseness).
All alternatives should have ___ (in relation to grammar)
All alternatives have correct spelling and grammar and grammatically “dovetail” with the stem.
item analysis forms a branch of
psychometrics
Item Analysis programs such
as “Classical Item Analysis”
and “Item Response Theory”
are vital for large-scale
testing because such
programs provide
statistical data regarding item by item response patterns among the
test-takers.
Such programs
provide analysis far beyond
simply indicating the
percentage of students who
selected the keyed answer.
what is the easiest way of calculating item difficulty?
find the proportion of students who answered the item correctly.
p=
of correct answers/# of students taking the rest
ranging from 0 to 1
for very Easy items: what is p is higher than?
0.85
More than 85% of the students answered the item correctly.
For Moderately Easy items: what is p between ?
0.65 to 0.75
65% to 75% of the students answered the item correctly.
for Difficult items what is p between?
0.40 to 0.50
40% to 50% of the students answered the item correctly.
for Very Difficult items: what is p less than?
less than 0.40
Less than 40% of the students answered the item correctly.
item discrimination refers to:
the degree to which an item can distinguish students with a low or high performance in the test based on their total test scores
item discrimination indicates whether an item is
is aligned with the rest of the items on the exam.
o (Is the item measuring the same construct as the other items?)
what does item discrimination range from?
-1 to +1
a higher number indicates a
higher discrimination (more it discriminates more likely to be green light question)
negative discrimination:
discrimination
between -1.0
and 0.0
No to modest discrimination:
Discrimination
between 0.0
and 0.
Highly discrimnating:
Discrimination
between 0.2
and 1.0
a good analysis program can helo reveal if an item you built is
techinally appropriate
Reliability ranges from
from 0 to 1
numbers closer
to 1.0 indicate (for reliability)
higher reliability
Low reliability numbers
are of concern regarding
the fairness of the
examination
what is reasonable
reliability for
a classroom
exam
.70 to .75
what is typical reliability for a diploma exam?
.85 to .93
