011_Measurement Reliability Flashcards
Reliability is the ___
consistency and stability of the results produced by a measure over time.
Consistency: The degree to which you can expect to get ___
the same results when measuring a variable/construct more than once on a single day.
Stability: The degree to which you can expect to get the___
same results when measuring a variable/construct over time (2 or more days).
The reliability coefficient is the __
proportion of observed differences in measures that are due to true differences in the attribute being measured.
observed score = true score + error
Can be stated in terms of variance – how much do the scores of a measure vary?
r= score of variance
Error
Potential sources for loss of reliability: • Participant • Environment • Procedure • Measure construction (wording) • Rating/Scoring (diffrent ppl.
Which term best describes the consistency of an instrument?
reliability
the true score plus error is = to____
the observed score
Methods of Estimating Reliability
Successive Measurements
• Test-retest
• Alternate (parallel) forms
• Alternate forms with a time interval
Single Measurement Point
• Internal consistency, split-halves
• Internal consistency, Cronbach’s alpha
• Internal consistency, Kuder-Richardson
Simultaneous Measurements (Multiple Researchers) • Interrater
Test-Retest
Method: One test given to same group twice following an intervening time interval; instrument scores are then correlated.
Advantages: • Easy method to use
• Only one instrument form is
needed
• Method is valid for speeded tests
Disadvantages:
• Requires two instrument administrations
• Difficult to control for memory and practice (not useful for many psychological tests, except for motor tests, sensory discrimination)
Alternate Forms
Method: Construct two forms of the same instrument (same number of items measuring similar content with similar level of difficulty) and administer to the same group in one administration; correlate the two sets of scores.
Advantages: • Indicates the degree to which the alternate forms give consistent results (gives an index of content sampling)
• Can control for memory
• Valid for speeded tests
Disadvantages: Need two equivalent forms of the instrument
Alternate Forms w/ Time Interval
Method: Construct two forms of the same instruments (same number of items measuring similar content with similar level of difficulty) and administer one form to a group in one administration, which is followed by the administration of the second form to the same group after an intervening time interval; correlate the two sets of scores.
Advantages: • Indicates the degree to which the alternate forms give consistent results (gives an index of content sampling) • Can control for memory • Valid for speeded tests • Well suited for measuring change in subjects over time.
Disadvantages: • Requires two instrument administrations
• Need two equivalent forms of the instrument
Simultaneous Measurements
Method: Two raters independently review and score participant performance (e.g., memory task performance). A correlation coefficient is computed for the two sets of scores and/or measures of agreement are computed (e.g., percent agreement).
Advantages
Allows for index of consistency for more subjectively scored phenomenon.
Disadvantages
• Must develop detailed scoring plans
• Must recruit and train raters
Reliability and Validity
Recall: It’s possible to have a reliable measure that is not valid.
It is possible to have a measure that is valid (produces valid results) but not reliable (produces inconsistent or unstable results)?
NO! If a measure has poor reliability, then there is a good chance that it is not measuring what it claims to measure. Interpretations and use of the results would be invalid.
the consistency of an instrument?
Reliability
