Test Construction

Question 1

Alternate Forms Reliability

Answer 1

coefficient of equivalence

Question 2

Coefficient Alpha (Chronback’s Alpha)

Answer 2

method for assessing internal consistency reliability when items are not answered dichotomously

Question 3

KR-20

Answer 3

method for assessing internal consistency reliability when items are answered dichotomously (they are either correct or not correct)

Question 4

Kappa Statistic

Answer 4

used to measure inter-rater reliability when data are nominal or ordinal (discontinuous)

Question 5

Test-Retest Reliability

Answer 5

yields a coefficient of stability

Question 6

Spearman Brown Formula

Answer 6

corrects for the artificially low reliability coefficient from testing via split-half reliability (low coefficient due to shorter test length)

Question 7

Size of reliability coefficient

Answer 7

smaller if it’s easy to get correct answer via random chance

Question 8

Difficulty Index

Answer 8

btwn 0 (no one can answer correct) - 1 (everyone answers correct)

Question 9

orthogonal factors v. oblique factors

Answer 9

orthogonal=uncorrelated (independent), oblique=correlated (dependent)

Question 10

Concurrant Validity

Answer 10

type of criterion-related validity. extent to which scores related to an external criterion

Question 11

Divergent (Discriminant) Validity

Answer 11

When scores on a measure are correlated with scores on unrelated traits (large coefficient) that is bad

Question 12

cross-validation

Answer 12

done during test revision, associated with “shrinkage” of the criterion-related validity

Question 13

external validity

Answer 13

researcher’s ability to generalize the results of the study to other individuals, settings, conditions

Question 14

internal validity

Answer 14

researcher’s ability to determine whether there is a causal relationship between variables

Question 15

pearson r

Answer 15

method of measuring inter-rater reliability, method for calculating criterion-related validity when both are on continuous scale

Question 16

methods of assessing internal consistency reliability

Answer 16

-split half (must correct with spearman brown) -KR-20 -chronback’s alpha

Question 17

4 methods of assessing reliability

Answer 17

inter-rater, internal consistency, alternate forms, test-retest

Question 18

standard error of measurement

Answer 18

the standard deviation of a theoretically normal distribution of test scores acquired by one individual on equivalent tests (related to the reliability coefficient and the SD of the test)

Question 19

calculating confidence interval of a true test score

Answer 19

person’s score + or - one or two standard errors of measurement (68% vs 95%)

Question 20

standard error of estimate

Answer 20

standard deviation of a theoretically normal distribution of criterion scores obtained by one person measured repeatedly

Question 21

Taylor-Russell Tables

Answer 21

numerically describe amount of improvement in decisions when a predictor is introduced

Question 22

incremental validity is optimized when

Answer 22

base rate is moderate (.5), and selection ratio is low

Question 23

item response theory

Answer 23

used to predict to what extent an examinee contains a certain trait based on response to a particular item

Question 24

factors affecting criterion-related validity

Answer 24

range of scores (more heterogenous testers means higher validity), reliability of the predictor, reliability of the predictor and criterion, criterion contamination (usually results in inflated validity)

Question 25

relationship between reliability of the predictor and criterion related validity

Answer 25

criterion-related validity cannot be higher than the square root of the reliability of the predictor

Question 26

percentile corresponding with one standard deviation above the mean, and 2 standard deviations above

Answer 26

84, 97

Question 27

type 1 error

Answer 27

the null hypothesis is falsely rejected

Question 28

Spearman rank order (rho) correlation coefficient

Answer 28

used when both variables are ranks

Question 29

phi correlation coefficient

Answer 29

used when both variables are true dichotomies

Question 30

biserial correlation coefficient

Answer 30

when one variable is continuous and one is an artificial dichotomy

Question 31

contingency correlation coefficient

Answer 31

when both variables are nominal

Question 32

when there is a moderator variable, make sure the test has

Answer 32

differential validity

Question 33

in a positively skewed distribution from greatest to lowest

Answer 33

mean, median, mode

Question 34

MST

Answer 34

“mean square total”=measure of treatment effects and error (MSB+MSW)

Question 35

MSW

Answer 35

“mean square within”=estimate of variability that is due purely to error

Question 36

MSB

Answer 36

“mean square between”=estimate of variability due to treatment effects plus error

Question 37

degrees of freedom for t-test for independent samples

Answer 37

N-2

Question 38

where item characteristic curve hits the y axis

Answer 38

probability of getting answer right by guessing

Question 39

statistical regression

Answer 39

tendency for extreme test scores to move toward the mean on retesting (threat to internal consistency if participants selected due to extreme scores on pre-test)

Question 40

purpose of rotation in factor analysis

Answer 40

makes pattern of factor loadings easier to interpret

Question 41

Solomon Four Group Design

Answer 41

controls for testing/test practice (which is a threat to internal validity)

Question 42

F ratio

Answer 42

mean square between groups divided by mean square within groups (MSB/MSW)

Question 43

eta

Answer 43

used to calculate correlation btwn x and y when relationship thought to be curvilinear