Lecture 2 - Reliability & Validity Flashcards
What is the fundamental equation of reliability theory? (Classical Testing Theory)
X = T + E
observed test score = true score + random error score
What does the equation mean?
σ_X^2 = σ_T^2 + σ_E^2
Across individuals, the observed-score variance is equal to the sum of true-score variance and error-score variance
What is ρ_XT^2 = σ_T^2 / σ_X^2?
The reliability coefficient - the proportion of observed score variance that is attributable to true-score variance.
If the test is perfectly reliable, ρ_XT^2 = σ_T^2 / σ_X^2 = 1
Reliability coefficient can also be expressed as
ρ_XT^2 = 1 - σ_E^2 / σ_X^2
What does Cronbach’s alpha measure?
Alpha is a measure of internal consistency, which refers to the interrelatedness of a set of items.
α = k/k-1 (1 - Σσ_i^2 / σ_X^2)
σ_i^2 = variance of the ith item σ_X^2 = variance of the test score X
It measures the proportion of the variance in the test scores that is attributable to shared or common variance
What are the caveats to using alpha?
- It is a function of test length
- It is only one form of reliability - concerned with only error variance arising from the sampling of items
- It is a lower bound of reliability
- It is persistently and incorrectly taken to be a measure of the internal structure (i.e., dimensionality) of the test
What is an acceptable level of alpha?
DeVellis suggested .70 while Clack and Watson argued for .80
The level of reliability depends on the context
What is the “new” definition for validity?
The degree to which evidence and theory support the interpretation of test scores entailed by the proposed uses of a test
What is needed for construct validation?
- Internal structure of a test - the way parts of a test are related to each other
- Association with other variables (criterion-related validity evidence)
- Association with other variables (construct validity evidence) - convergent and discriminant evidence
