Test Construction

Question 1

Item difficulty

Answer 1

• Measured using an item difficulty index ranging from 0 - 1
• Equation = Total number of examinees passing the exam divided by total number of examinees
• Optimal difficulty level depends on likelihood of answering correctly by chance, the goal of the testing, etc.

Question 2

Item discrimination

Answer 2

• The extent to which an item differentiates between examinees who obtain high versus low scores on the entire test
• D = U - L
• Ranges from -1 to +1
• Items with a discrimination index of .35 or higher is typically acceptable

Question 3

Item characteristic curve

Answer 3

• Constructed for each item
• Plot the proportion of examinees in the sample who answered correctly against the total test score, performance on an external criterion, or an estimate of the latent ability or trait measured by the item

Question 4

Item response theory

Answer 4

Is sample invariant

Question 5

Classical test theory

Answer 5

Uses 2 methods of item analysis: item difficulty and item discrimination

Question 6

Limitations of CTT

Answer 6

• Item and test parameters are sample dependent
• Difficult to equate scores across content

Question 7

Item’s level of difficulty

Answer 7

Ability level at which 50% of the examinees provide a correct response

Question 8

Item’s ability to discriminate

Answer 8

Indicated by the slope of the curve
The steeper the slope, the greater the discrimination

Question 9

Probability of guessing correctly

Answer 9

Indicated by the point at which the ICC intercepts the vertical axis

Question 10

Test score in Classical Test Theory

Answer 10

X = T + E
T = True score component
E = Error component (measurement error)

Question 11

Reliability coefficient

Answer 11

Ranges from 0 to 1
Correlation coefficient
Unlike most correlations, the r is never squared
Ex. a reliability coefficient of .89 means that 89% of variability in obtained scores is true score variability

Question 12

Test-retest reliability

Answer 12

Same test to same group of examinees on two different occasions
Coefficient indicates stability/consistency
May be impacted by a PRACTICE EFFECT

Question 13

Alternate forms reliability

Answer 13

Two forms of the same test are administered at the same time point
Coefficient of equivalence
May be impacted by CONTENT SAMPLING
Good for speeded tests

Question 14

Split-half reliability

Answer 14

Scores on two halves of the test are correlated (e.g., odd versus even numbered items)
Usually underestimates a test’s true reliability

Question 15

Spearman-Brown Prophecy Formula

Answer 15

Used to correct the split-half reliability coefficient
Provides an estimate of the reliability coefficient in a given test length
Tends to overestimate a tests true reliability

Question 16

Cronbach’s coefficient Alpha

Answer 16

Calculates the average reliability from all possible splits of the test
This is a conservative measurement of reliability

Question 17

Kuder-Richardson Formula 20

Answer 17

Used when test items are scored dichotomously

Question 18

Kappa statistic/Cohen’s Kappa

Answer 18

Used when scores or ratings represent a nominal or ordinal scale of measurement

Question 19

Test length

Answer 19

The longer the test, the less the relative effects of measurement error and the larger the reliability coefficient
Spearman Brown prophecy formula can also be used to estimate the impact of lengthening a test

Question 20

Range of scores

Answer 20

The reliability coefficient is maximized then the range of scores is unrestricted

Question 21

Standard error of measurement

Answer 21

Index of the amount of error that can be expected in obtained scores due to the unreliability of the test
SEM = SD sqrt(1-reliability coefficient)

Question 22

Internal consistency reliability

Answer 22

How well items within a test correlate with other items on the same test
This includes split-half reliability and Cronbach’s coefficient alpha

Question 23

Content sampling

Answer 23

Impacts split-half reliability and coefficient alpha

Question 24

Inter-rater reliability

Answer 24

Measured using a kappa or a percent agreement

Question 25

Kendall's coefficient of concordance

Answer 25

Used to assess inter-rater reliability when three or more raters and ratings are reported as ranks

Question 26

Consensual observer drift

Answer 26

When two or more observers working together influence each other's ratings and both assign ratings in a similarly idiosyncratic way

Question 27

Impact of guessing on reliability

Answer 27

As the probability of guessing correctly increases, the reliability coefficient decreases

Question 28

Content validity

Answer 28

Items on the test adequately represent the domain being measured

Question 29

Construct validity

Answer 29

The test has expected relationships with other variables

Question 30

Convergent and Discriminant Validity

Answer 30

Convergent validity = high correlations ith measures of the same and related traits Discriminant validity = low correlations with measures of unrelated characteristics

Question 31

Multitrait-Multimethod Matrix

Answer 31

Systematically organizes data collected when assessing a test's convergent and discriminant validity Includes coefficients that are: monotrait-monomethod, monotrait-heteromethod, heterotrait-monomethod, heterotrait-heteromethod

Question 32

Criterion-related Validity

Answer 32

Test correlates/predicts an examinee's performance on some external criterion

Question 33

Standard error of estimate

Answer 33

SEE = Standard deviation of criterion scores sqrt(1-validity coefficient squared) Used to construct a confidence interval around an estimated score

Question 34

Incremental validity

Answer 34

The increase in correct decisions that can be expected if the predictor is used as a decision making tool Positive hit rate - base rate

Question 35

Specificity and sensitivity

Answer 35

Provide information about a predictor's accuracy when administered to a group of individuals who are known to have a disorder of interest Sensitivity = % of people ho have the disorder and were accurately identified by true positives and false negatives Specificity = % of people who do not have the disorder and were accurately identified by true negatives and false positives

Question 36

Criterion Contamination

Question 37

Cross-valdiation

Question 38

Shrinkage

Question 39

Concurrent validity

Answer 39

A form of criterion-related validity. When criterion data are collected prior to or at the same time as data on the predictor

Question 40

Predictive validity

Answer 40

When the criterion is measured at some point after the predictor has been administered

Question 41

Criterion related validity coefficient

Answer 41

Ranges from -1 to 1

Question 42

Positive predictive value

Answer 42

Probability that people ho test positive have the disorder

Question 43

Positive likelihood ratio

Answer 43

The extent to which a positive result affects the probability that the person has a disorder A useful predictor should have an LR+ of at least 1.0

Question 44

Relationship between reliability and validity

Answer 44

A test's reliability always places a ceiling on its validity High reliability however does not guarantee validity