Test Construction

Question 1

Standard Error of Measurement/CI

Answer 1

The SEM is used to construct the CI around a specific test score. Depends on test’s SD and reliability coefficient.

Question 2

Criterion Contamination

Answer 2

Bias introduced to criterion score as a result of person’s knowledge about their performance on the predictor

Question 3

Item Difficulty

Answer 3

Determined by dividing # of people who got it right by total #. 0 is very difficult, 1.0 is easy. .5 difficulty is preferred.

Question 4

Reliability

Answer 4

Consistency of test scores over time, across forms, or across items. Can do test-retest, coefficient alpha, interrater, split-half, alternative forms. Reliability of .80 means 80% of variability is TRUE variability.

Question 5

Cross-validation and shrinkage

Answer 5

Re-assess criterion-related validity on a new sample to see how generalizable coefficient is. Coefficient shrinks as a result because the “chance factors” operating in original sample aren’t present.

Question 6

Standard Error of Estimate/CI

Answer 6

Index of error when predicting criterion scores. Used to make a CI around a predicted score. Magnitude depends on criterion’s SD and validity coefficient.

Question 7

Classical Test Theory

Answer 7

Observed variability in test scores reflects: 1) true differences between examinees on the attribute, and 2) effects of random error.

Question 8

Factor Analysis

Answer 8

Stat technique used to determine how many factors are needed to account for intercorrelations among a set of tests, substests, or test items.

Question 9

Construct Validity

Answer 9

Extent to which a test measures a hypothetical trait it is intended to measure.

Question 10

Incremental Validity

Answer 10

Extent to which predictor increases decision-making accuracy. Calculate by subtracting base rate from positive hit rate. (linked to true and false negatives and positives, and criterion cut-off scores)

Question 11

Kappa stat

Answer 11

Correlation coefficient used to assess interrater reliability

Question 12

Split half/ Spearman-Brown

Answer 12

Split-half: split test in half and correlate two halves. Tends to underestimate reliability
Spearman-Brown: corrects split-half technique and figures out reliability if test were full length.

Question 13

Item discrimination

Answer 13

the extent to which a test differentiates between examinees who obtain high versus low scores on the test or on an external criterion. Ranges from -1.0 to +1.0. If all in upper group and none from lower get it right, score is +1.0

Question 14

Coefficient Alpha/KR-20

Answer 14

Both are used to assess internal consistency reliability (inter-item consistency). KR-20 used for test items that are scored dichotomously.

Question 15

Sensitivity and Specificity

Answer 15

Sensitivity= % of people in sample who have disorder and were accurately identified

Specificity: % of people who do not have disorder and were accurately identified as NOT having it

Question 16

Factor loadings and communality

Answer 16

Factor loading= correlation between a test or other variable, when squared this is the amount of variability in test accounted for by that factor.
Communality= total variability in scores accounted for by the factor analysis.

Question 17

Oblique and Orthogonal Rotation

Answer 17

Oblique= rotation produces correlated factors

Orthogonal= rotation produces uncorrelated factors

Rotation is done to simplify interpretation of factors

Question 18

Content validity

Answer 18

Extent to which test adequately samples the domain of info or skill (expert judgment)

Question 19

Test length/ range of scores

Answer 19

Increasing test length with more items of similar content and quality increases reliability. Or increase heterogeneity of sample in terms of the attribute measures, which increases the range.

Question 20

Relationship between reliability and validity

Answer 20

reliability is a necessary but insufficient condition for validity!

Question 21

Item characteristic curve

Answer 21

Constructed in item response theory for each item. Provides info on relationship between examinee’s level on the ability or trait measures and the probability of responding correctly.