Test Construction Flashcards

1
Q

Alternate Forms Reliability

A

coefficient of equivalence

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Coefficient Alpha (Chronback’s Alpha)

A

method for assessing internal consistency reliability when items are not answered dichotomously

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

KR-20

A

method for assessing internal consistency reliability when items are answered dichotomously (they are either correct or not correct)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Kappa Statistic

A

used to measure inter-rater reliability when data are nominal or ordinal (discontinuous)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Test-Retest Reliability

A

yields a coefficient of stability

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Spearman Brown Formula

A

corrects for the artificially low reliability coefficient from testing via split-half reliability (low coefficient due to shorter test length)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Size of reliability coefficient

A

smaller if it’s easy to get correct answer via random chance

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Difficulty Index

A

btwn 0 (no one can answer correct) - 1 (everyone answers correct)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

orthogonal factors v. oblique factors

A

orthogonal=uncorrelated (independent), oblique=correlated (dependent)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Concurrant Validity

A

type of criterion-related validity. extent to which scores related to an external criterion

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Divergent (Discriminant) Validity

A

When scores on a measure are correlated with scores on unrelated traits (large coefficient) that is bad

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

cross-validation

A

done during test revision, associated with “shrinkage” of the criterion-related validity

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

external validity

A

researcher’s ability to generalize the results of the study to other individuals, settings, conditions

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

internal validity

A

researcher’s ability to determine whether there is a causal relationship between variables

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

pearson r

A

method of measuring inter-rater reliability, method for calculating criterion-related validity when both are on continuous scale

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

methods of assessing internal consistency reliability

A

-split half (must correct with spearman brown) -KR-20 -chronback’s alpha

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

4 methods of assessing reliability

A

inter-rater, internal consistency, alternate forms, test-retest

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

standard error of measurement

A

the standard deviation of a theoretically normal distribution of test scores acquired by one individual on equivalent tests (related to the reliability coefficient and the SD of the test)

19
Q

calculating confidence interval of a true test score

A

person’s score + or - one or two standard errors of measurement (68% vs 95%)

20
Q

standard error of estimate

A

standard deviation of a theoretically normal distribution of criterion scores obtained by one person measured repeatedly

21
Q

Taylor-Russell Tables

A

numerically describe amount of improvement in decisions when a predictor is introduced

22
Q

incremental validity is optimized when

A

base rate is moderate (.5), and selection ratio is low

23
Q

item response theory

A

used to predict to what extent an examinee contains a certain trait based on response to a particular item

24
Q

factors affecting criterion-related validity

A

range of scores (more heterogenous testers means higher validity), reliability of the predictor, reliability of the predictor and criterion, criterion contamination (usually results in inflated validity)

25
relationship between reliability of the predictor and criterion related validity
criterion-related validity cannot be higher than the square root of the reliability of the predictor
26
percentile corresponding with one standard deviation above the mean, and 2 standard deviations above
84, 97
27
type 1 error
the null hypothesis is falsely rejected
28
Spearman rank order (rho) correlation coefficient
used when both variables are ranks
29
phi correlation coefficient
used when both variables are true dichotomies
30
biserial correlation coefficient
when one variable is continuous and one is an artificial dichotomy
31
contingency correlation coefficient
when both variables are nominal
32
when there is a moderator variable, make sure the test has
differential validity
33
in a positively skewed distribution from greatest to lowest
mean, median, mode
34
MST
"mean square total"=measure of treatment effects and error (MSB+MSW)
35
MSW
"mean square within"=estimate of variability that is due purely to error
36
MSB
"mean square between"=estimate of variability due to treatment effects plus error
37
degrees of freedom for t-test for independent samples
N-2
38
where item characteristic curve hits the y axis
probability of getting answer right by guessing
39
statistical regression
tendency for extreme test scores to move toward the mean on retesting (threat to internal consistency if participants selected due to extreme scores on pre-test)
40
purpose of rotation in factor analysis
makes pattern of factor loadings easier to interpret
41
Solomon Four Group Design
controls for testing/test practice (which is a threat to internal validity)
42
F ratio
mean square between groups divided by mean square within groups (MSB/MSW)
43
eta
used to calculate correlation btwn x and y when relationship thought to be curvilinear