Midterm 1 - Reliability (Ch. 4)

Question 1

(reliability/validity) is a condition for (reliability/validity)

Answer 1

reliability is a condition for validity

Question 2

reliability means ____ of scores

Answer 2

consistency

Question 3

What are 3 assumptions of Classical Test Theory about measurement?

Answer 3

• no measurement instrument is perfectly reliable (observed score = true score + error)
• measurement errors are random (mean error = 0)
• measurement errors are normally distributed

Question 4

(random/systematic) error lowers reliability of a test

Answer 4

random!

Question 5

(T/F) systematic measurement errors lower reliability of a test over time

Answer 5

FALSE, does not lower reliability bc test is inaccurate by same amount every time

Question 6

What impact do random and systematic errors have on the average of a distribution?

Answer 6

• random: no effect on average, just variability around it
• systematic: changes average (bias)

Question 7

Classical test theory assumes that the variance of observed scores = ____ + _____

Answer 7

variance of true scores + error variance (random error)

σX(squared) = σT(squared) + σE(squared)

Question 8

The reliability coefficient (rxx’) measures the proportion of observed test scores accounted for by ____

Answer 8

variability in true scores

Question 9

What is the formula for the reliability coefficient?

Answer 9

rxx’ = true score variance/observed score variance

rxx’ = σT(squared)/σX(squared)

Question 10

the reliability coefficient is always between __ and __

Answer 10

0 and 1

Question 11

Classical test theory assumes that two tests are parallel if:
1.
2.
3.

Answer 11

1. true scores are the same (which means equal observed score means)
2. equal error variance
3. same correlations w other tests

Question 12

Standard error of measurement (SEM) indicates the amount of _____ expected in an individual’s (true/observed) test score. SEM corresponds to _____

Answer 12

uncertainty or error; observed test score
corresponds to SD of distribution of scores one would obtain by repeatedly testing a person

Question 13

Identify each of the variables in this equation:
SEM = SDx * √(1-rxx’)

if SDx is high, SEM will be (higher/lower)
if rxx’ is high, SEM will be (higher/lower)

Answer 13

SEM = standard error of measurement
SDx = standard deviation of observed scores
rxx’ = reliability of observed scores

if SDx is high, SEM will be HIGHER
if rxx’ is high, SEM will be LOWER

Question 14

Confidence interval (CI) is a range of scores that _____

Answer 14

we feel confident will include the true score

Question 15

Identify each of the variables in the following formula:
CI = X +/- (zCI)(SEM)

Answer 15

CI = confidence interval
X = observed score
zCI = z of CI’s boundaries
SEM = standard error of measurement

Question 16

what is the z score (zCI) for a 95% confidence interval?

Answer 16

1.96

Question 17

What is the 95% confidence interval for an observed score of 110 with a SEM of 5?

Answer 17

CI = X +/- (zCI)(SEM)
CI = 110 +/- 1.96(5)
CI = 110 +/- 9.8
CI = approx 100 and 120

Question 18

Reliability of a test can be increased by ____

Answer 18

adding items! (as long as they’re valid)

Question 19

What does the Spearman-Brown formula do?

Answer 19

predicts effect of lengthening or shortening a test on reliability

Question 20

Identify the variables in the Spearman-Brown formula:

Predicted rxx’ = [n(rxx’)] / [1+(n-1)rxx’]

Answer 20

Predicted rxx’ = reliability of observed scores
n = degree of increase/decrease in test length (ex double = 2; half = .5)

Question 21

What are the 3 main types of reliability?

Answer 21

• test-retest
• internal consistency
• interrater agreement

Question 22

What is test-retest reliability?

Answer 22

• coefficient of stability
• admin same test to same group @ diff occasions and correlate both sets of scores

Question 23

(T/F) when evaluating reliability we always want test-retest reliability to be high

Answer 23

FALSE - should match construct (should be lower when measuring a construct we expect to change!)

Question 24

What is another form of test-retest reliability?

Answer 24

• coefficient of equivalence
• make two similar versions of a test and admin both to same group in very short period, then correlate scores

Question 25

What is internal consistency?

Answer 25

• look at how consistently examinees performed across items or subsets of items
• one test administration!

Question 26

What are the two methods we discussed for assessing internal consistency?

Answer 26

• Split half methods (older): correlate scores based on first half or second half of items OR based on odd/even items
• contemporary approach: Cronbach’s alpha (avg of all possible split-half correlations); unaffected by how items are arranged in test

Question 27

(T/F) we always want internal consistency coefficients to be as high as possible

Answer 27

FALSE
- higher for narrow constructs (ex sadness)
- lower for broader constructs (ex neuroticism)
**very high scores can indicate insufficient sampling of the domain!

Question 28

What is interrater reliability?

Answer 28

• have diff observers rate a certain behaviour
• look @ level of agreement btw observers

Question 29

Identify the variables in the Kappa formula for 2 categories
k = (po - pe) / (1 - pe)

This formula assesses ____
k ranges from __ to __ with higher values indicating ___

Answer 29

k = kappa (interrater agreement corrected by chance)
po = observed proportions (how often raters agreed out of all observations)
pe = expected proportions

assesses interrater agreement corrected by chance
k ranges from -1 to 1; higher = higher agreement

Question 30

What is one challenge of the interrater reliability method?

Answer 30

• can be hard to precisely define a construct

Question 31

What is the main source of error for each of these forms of reliability coefficients?
- test-retest
- parallel forms
- internal consistency
- interrater

Answer 31

• test-retest: time sampling (length of test-retest interval)
• parallel forms: item sampling (homogeneity of items)
• internal consistency: item sampling (homogeneity, length)
• interrater: observer differences