Test Construction

Question 1

Item Analysis

Answer 1

Determine which items to retain in final test

Question 2

Item Difficulty Index (p)

ranges 0 to 1 - 0.5 diff. level preferred

Answer 2

of exminees ans correct/total examinees

Question 3

Item Discrimination (D)
(ranges from -1 to 1)

Answer 3

%examinees in upper scoring grp -%examinees in lower scoring grp

Question 4

Reliability Coefficient (RC)

Answer 4

• Estimates tests reliability (variability)
• Ranges from 0 to 1
• .91 RC=91% due to true score variability & 9% due to measurement error

Question 5

Methods for Estimating Reliability

Answer 5

• Test-retest (coefficient of stability)
• Alternate forms (coeff. of equivalence)
• Split-half (internal consistency reliability)
• Spearman-Brown used w/^ (determine test’s true reliability)
• Coeff. Alpha (inter item not 2 halves)
• Kuder-Richardson-substitute for co. alpha when items scored dichotomously
• Inter-rater (when scored subjectively)
• Coeff. of concordance (interrater & ranks)

Question 6

Spearman-Brown Formula

Answer 6

• Estimate effects of lengthening or shortening a test on reliability coeff.

Question 7

Std Error of Measurement (SEM)

Answer 7

• How much an individual’s obtained score reflects his/her true score
• std deviation of test scores x sq root of 1 minus reliability coeff.

Question 8

Validity

Answer 8

A test is valid when it accurately measures what it is designed to measure

Question 9

Content Validity

Answer 9

When test will be used to measure one or more content/behavior domains

Question 10

Construct Validity

Answer 10

When test will be used to measure hypothetical trait (construct) e.g. achievement, intelligence or mechanical aptitude

Question 11

Criterion-related Validity

Answer 11

When a test will be used to estimate or predict performance on another measure

Question 12

Construct Validity

• Convergent
• Divergent

Answer 12

Convergent - high correlations w/ measures that assess the same construct
Divergent - low correlations w/ measures of unrelated characteristics (=discriminant validity)

Question 13

Multi-trait Multi-Method Matrix (MTMM)

Answer 13

Convergent and Discriminant Validity

• Monotrait-heteromethod large: Convergent validity
• Heterotrait-monomethod & heterotrait heteromethod small: Discriminant validity

Question 14

Factor Analysis

Answer 14

• determine construct validity
• factor matrix
• factor loading (shared variability sq. coeff)
• Test has 0.5 correlation with Factor 1 = 25% of variability in test scores is explained by Factor 1

Question 15

Orthogonal Factors (unrelated)

Answer 15

• communality calculated by summing the factor loadings

- Factor 1=.50 Factor 2=.20 (communality=.29)

Question 16

Sensitivity

Answer 16

Probability that predictor will correctly id pple w/disorder

formula: true +ves plus false -ves

Question 17

Decision-Making Accuracy

Answer 17

Positive hit rate - Base rate

Question 18

Predictor cut-off (PC)

Answer 18

Determines if someone is positive or negative (to reduce false positives, raise PC)

Question 19

Criterion cut-off (CC)

Answer 19

Determines if he/she is true or false (reduce false positives lower CC)

Question 20

Criterion Contamination

Answer 20

Occurs when knowledge of a person’s predictor performance affects how he/she rates person on criterion

Question 21

Correction for Attenuation formula

Answer 21

Determines impact of increasing reliability of the predictor (test) &/ the criterion on the predictor’s validity

Question 22

Norm-Referenced Interpretation

Answer 22

• percentile rank (% in the normal sample who obtained low scores)
• standard scores (examinees scores in terms of std deviation from the mean)

Question 23

Standard Scores

Answer 23

Z-Score

• mean of 0
• std deviation of 1
• e.g. z score of -1 (score raw score is one std deviation below the mean)

Question 24

Criterion-referenced Interpretation

Answer 24

Percent Scores

- indicate proportion of test content (% of test items) examinees answered correctly