Ch. 8 - Test Development

Question 1

5 stages of test development

Answer 1

1. conceptualization 2. construction 3. tryout 4. item analysis 5. test revision

Question 2

test construction

Answer 2

process of writing possible test items

Question 3

test tryout

Answer 3

administering a test to a representative sample of testtakers under conditions that simulate those of the final version of the test

Question 4

some questions to ask when developing a new test

Answer 4

What is the test designed to measure? (what construct)
Is there a need for this test?
Who will use and take the test?
How will the test be administered?
Is there any potential for harm?
How will meaning be attributed to the scores on the test?

Question 5

on a norm-referenced test, a good item is one that…

Answer 5

high scorers on the whole test get right

Question 6

on a criterion referenced test, you need to do exploratory/pilot work with…

Answer 6

a group known to have mastered the skill

Question 7

pilot work/study

Why is it done?

Answer 7

work done surrounding the creation of the prototype of a test
done to determine how to best measure a targeted construct

Question 8

scaling

Answer 8

setting rules for assigned #s in measurement; the process by which a measuring device is designed and calibrated and by which #s (or other indices) AKA scale values are assigned to different amounts of the thing being measured.

Question 9

stanine scale

Answer 9

all raw scores on the test can be transformed into scores that range from 1 to 9

Question 10

age and grade-based scales

Answer 10

if testtakers’ performance is a function of age or grade is of critical interest

Question 11

Likert scale

Answer 11

very reliable, has a scale of 1-5 or 1-7

Question 12

rating scales

provide what kind of data?

Answer 12

grouping of words, statements, or symbols on which judgments of the strength of a particular thing are indicated by the testtaker
ALL rating scales provide ordinal data

Question 13

method of paired comparisons

Answer 13

testtakers are presented with a pair of stimuli and must choose between then.
provide ordinal data

Question 14

comparative scaling

Answer 14

testtaker must judge a stimulus in comparison with every other stimulus on the scale

Question 15

catagorical scaling

Answer 15

stimului are placed into one of two or more alternative cateogires that differ quantitatively with respect to some continuum. For ex: sort into “never justified” “sometimes justified” “always justified”

Question 16

Guttman scale

Answer 16

items on it range sequentially from weaker to stronger expressions of the thing. (everyone who agrees with the stronger statement agrees with the weaker ones). used in consumer research
AKA scalogram analysis

Question 17

direct estimation vs indirect estimation

Answer 17

in direct estimation, you don’t need to transform a testtaker’s responses into some other scale. in indirect, you do need to transform those responses.

Question 18

equal-appearing intervals method

Answer 18

the only rating scale described that has items that are interval in nature (ex: suicide scale) - there are presumed to be equal distances between the values on the scale (interval scale)

Question 19

How many test items should an item pool contain for a multiple-choice test?

Answer 19

twice the number of the final number of test items

Question 20

item pool

Answer 20

assembly of many items (from brainstorming all possibilities or many possibilities of test items)

Question 21

selected-response format (item types)

Answer 21

multiple choice
matching
true-false (binary-choice item)

Question 22

constructed-response format (item types)

Answer 22

completion item
short answer
essay
looking for synthesis of info

Question 23

item bank

Answer 23

collection of GOOD test items. These items will continue to be selected and used or rotated. Finalized version of the item bank.

Question 24

CAT

Answer 24

computerized adaptive testing - test-taking process wherein items presented to the testtaker are based on performance of previous items. may be displayed according to rules (e.g. only after you get 2 hard ones right, show the next level).

Question 25

floor effect vs. ceiling effect reduced by?

Answer 25

CAT tends to reduce these. floor effect - not distinguishing between low scores/low ability
ceiling effect - not distinguishing between high scorers/high ability

Question 26

item branching

Answer 26

the ability of a computer to tailor the content and order of presentation items on the basis of responses to previous items

Question 27

CAT found to reduce…

Answer 27

# of test items (by 50%)
measurement error (by 50%)

Question 28

class scoring AKA

Answer 28

category scoring
testtakers will be placed in a certain group/class with other testtakers whose pattern of responses was similar in some way (e.g., diganosis)

Question 29

cumulative scoring model

Answer 29

higher the score, the higher the testtaker is on the thing being measured

Question 30

ipsative scoring

Answer 30

compares testtakers score on one thing within a test to another thing on a test (thing = scale). comparing yourself with yourself.
e.g. Jon is cooler than he is smart BUT you can’t say Jon is cooler than Jenny

Question 31

test tryout - how many people?

Answer 31

no fewer than 5 for EACH item on the test. more the better. preferably 10.

Question 32

How do you tell whether a test item is good?

Answer 32

item analysis

generally, a good test item is answered correctly by high scorers on the test as a whole

Question 33

item analysis

Answer 33

the statistical scrutiny of test data

Question 34

what do you scritinize in item analysis?

Answer 34

item’s: difficulty, validity, relaibility, item discrimination (IDDRV)

Question 35

item-difficulty index

expected values

Answer 35

denoted by lowercase italicized p
= to testtakers who got it correct / total testtakers
greater the p, the easier the item
if >.90 - is it really needed? other than as a giveaway/warmup for those with test anxiety

Question 36

what is an item-difficulty index called on a personality test?

Answer 36

item-endorsement index

Question 37

item-reliability index

Answer 37

shows the internal consistency of a test
higher the value, the greater the test’s internal consistency
=s*r (item SD * correlation between item score and test score)

Question 38

item-validity index

Answer 38

measures the degree to which a test is measuring what it’s supposed to measure

Question 39

item-discrimination index

Answer 39

measures how adequately an item separates/discriminates between high and low scores on the entire test
yields a lowercase italicized d
d compares performance on a particular item with performance on the upper and lower regions of a distribution of continuous test scores
higher the d, the higher the # of high scorers are answering it correctly

Question 40

what does a high d value mean

Answer 40

higher the d, the higher the # of high scorers are answering a test item correctly
Bonus: if it’s a -d, that means that more low-scorers answer it correctly than high scorers
Bonus: if d = 0, same # of high and low scorers get it right

Question 41

what does a high p mean

Answer 41

the test item is easy

Question 42

analysis of item alternatives

Answer 42

for multiple-choice items, see how many people answered the distractors and evaluate them appropriately (e.g., maybe too distracting/wording needs to be changed)

Question 43

item-characteristic curve

Answer 43

ICC - a graphic representation of item difficulty and discrimination

Question 44

steep slope in ICC means?

Answer 44

greater item discrimination

Question 45

“good” item looks like what in ICC

Answer 45

straight line with a slope

Question 46

“good” item for a cutoff-score test or criterion-based test

Answer 46

looks like the top of an F

Question 47

guessing - what issues does it present in item analysis?

Answer 47

• guesses are not made totally randomly
• how do we deal with omitted items?
• some people are luckier guessers than others

Question 48

item fairness

Answer 48

the degree (if any) a test item is biased

Question 49

biased test item

Answer 49

favor one particular group of examinees when differences in group ability are controlled

Question 50

if an item is fair, its ICC should…

Answer 50

not be significantly differnt for different groups regardless of ability

Question 51

item analysis in speed tests

Answer 51

yield misleading or uninterpretable results because items that are closer to the end appear more difficult just because few people were able to finish them

Question 52

what are methods of qualitative item analysis

Answer 52

intervies, group discussions, “think aloud” test administration (sheds light on thought patterns), and sensitivity reviews

Question 53

sensitivity reviews

Answer 53

an expert panel. items on a test are examined for fairness to all prospective testatkers, flag offensive language, stereotypes

Question 54

test revision (as a stage in test development) - strategy

Answer 54

characterize each item according to its strengths and weaknesses
consider the purpose of the test - if for hiring and firing, eliminate biased items
if for culling most skilled performers - get items with the best item discrimination to ID the best of the best

Question 55

sandardization

Answer 55

process used to introduce objectivity and uniformity into test administration, scoring, and interpretation

Question 56

What do you need to do after item analysis?

Answer 56

administer the revised test under standardized conditions, then cross-validation

Question 57

When should you revise a test?

Answer 57

stimulus look dated, dated vocabulary, offensive language, test norms aren’t adequate (group membership change), age-related shifts in the abilities over time, improve the reliability or validity of the test, theory on which test was based has improved

Question 58

steps to revising an existing test

Answer 58

-all steps to make a new one (conceptualization, construction, tryout, item analysis, revision) + need to determine whether there is equivalence between the old and new versions of the test. likely scores will not mean the same thing (item analysis to evaluate stability of items between revisions of the same test)

Question 59

cross-validation

what is inevitable?

Answer 59

re-validation of a test on a sample of testtakers other than the original group the test was found to be valid on. (aCROSS groups)
validity shrinkage is inevitable

Question 60

co-validation

Answer 60

test validation process conducted on two or more tests using the same sample of testtakers (economical - test subjects ID’ed once, personnel costs)

Question 61

co-norming

benefits?

Answer 61

co-validation on two tests and creating norms or revising existing norms
good for test users if tests are often used together bc they are normed on the same population (sampling error has been eliminated basically)
like co-validation, saves money

Question 62

quality assurance in test revision

Answer 62

confirming that a test is given the same way

Question 63

anchor protocol

Answer 63

test protocol scored by a highly trained scorer, designed as a model for scoring and mechanism for resolving scoring discrepancies

Question 64

protocol drift

Answer 64

the discrepancy between and anchor protocol and another scorer’s protocol

Question 65

differential item functioning

Answer 65

(DIF) - when an item functions differently in one group of testtakers as compared to another group of testtakers known to have the same/similar level of underlying trait. This means that for some reason respondents from different groups have different probabilities of endorsing as a function of their group membership (ex: Asian women fear cultural shame around feeling depressed)