quiz 5: reliability part one Flashcards
why is reliability important
- Ensures we are measuring something meaningful
- Random measurement error (noise) is always present
what is error
-Inevitable
- Occurs when measurement of a construct is confounded by factors that are not relevant to the construct we want to measure
- Motivation or lack of effort when measuring intellectual ability
- Reading comprehension when measuring quantitative ability
what is systematic error
- Systematic Error = a “mistake” that can be corrected and eliminated
- Mistakenly keyed correct answers
- Lack of familiarity with scoring criteria
what is random error
- Impossible to eliminate
- Inherent in any measurement attempt
- “Random” means it is NOT correlated with the obtained scores
- Therefore, it is impossible to measure (or eliminate)
what is observed score
- (obtained, measured) score = score we obtain whenever we administer a test
- The person’s measured standing on the trait we are interested in measuring
what is true score
- what the person’s score would be if there were no measurement error
- The person’s actual standing on the trait or the actual “amount” of the trait they possess.
are observed scores and true scores the same
Because error is inevitable, the observed score might not be (usually isn’t) the same as the true score
what is classic reliability theory
- If Xo is the observed score
- Xt is the true score, and
- e is the amount of error, then
- Xo = Xt + e*
what are the implications of error (4 of them)
1) Error cannot be measured (1)
- Error scores are RANDOM
- Uncorrelated with either observed or true scores
- Error affects a person’s observed score in ways that are independent of his/her true score
2) Error tends to cancel out across respondents (2)
- Inflates the scores of some and deflates the scores of others
- Average effect of error across respondents is zero i.e.
- This means that the mean of the observed scores is equal to the mean of the true score Xo = Xt
3) Since error is random, i.e., uncorrelated with either the true score or the obtained score, then the following also applies: (3)
- s2o = s2t + s2e (variance of the observed score, variance of the true score, error variance – can never be zero)
- NOTE: Variance of X+Y = Variance of X plus Variance of Y plus Covariance of X and Y. If X and Y are not correlated, then Cov (X,Y) = 0.
4) Variance of the observed scores is always larger than the variance of the true scores (4)
- Error makes people look more different from one another than they actually are
how to get reliability equation
on paper
what does reliability in words mean
- If the reliability of a test is .95, then 95% of the variability in obtained scores is due to actual (real, true) differences among test-takers on the trait (construct) being measured
- 95% of the differences we see among the observed scores of the test takers can be attributed to differences in their true levels on the trait being measured
explain error variance
The term on the right side of the + sign tells us the proportion of variance in observed scores that is due to error variance
-If the reliability = .95, then 5% of the variance in observed scores is due to error variance
what part of the equation is error variance
on paper
implications of reliability
- Reliability is a theoretical property of a test and cannot be computed directly
- It can only be estimated from real data
- There is no single method that provides completely accurate estimates of reliability under all conditions
- We can never calculate “the” reliability of a test
- Instead, we can calculate the effect on a test’s reliability of different sources of error
typical sources of measurement error
- Time Sampling (when the test is given)
- Time of the day a person takes a test
- Item Sampling (which items were selected)
- Internal Consistency (whether the items are all measuring the trait of interest)
- Inter-rater Differences (whether different raters assign the same score)
