Ch 5.2 Reliability

Question 1

Consistency Across Time: Test-Retest Reliability

Answer 1

• Correlating paires of scores from the same people on two different administrators of the same test
• Appropriate for relatively stable constructs (personality)

Question 2

Consider

Answer 2

• Time between administrations

– Generally, as time increases reliability decreases

– Coefficient of stability (interval between testing > 6 mos.)

• Intervening factors

– Going through treatment?

– Learning occurring?

When a change is assumed we do not use test-retest reliability

Question 3

Consistency Across Raters: Inter - Scorer (Rater) Reliability

Answer 3

• Degree of agreement of consistency between two or more scorers
• Commonly included in test manuals
• High consistency indicates that with sufficient training, scores may be derived in a systematic, consistent way by various scorers

Consider

• Clarity of scoring criteria

Question 4

Summary: Different Approaches to Reliability [placeholder]

Answer 4

.

Question 5

Self-Reports

Answer 5

• Test-Retest
• Alternate Form/Parallel Form
• Internal Consistency

Question 6

Interviews/Behavioral Observations

Answer 6

• Inter-Rater Reliability

Question 7

How reliable is reliable?

Answer 7

Depends on the purpose of the test & importance of results

• .65 - .70 (minimal_
• .80+ (good)
• .90+ (really good, sometimes too good)
• Possible item redundancy

Question 8

Interpretation Considerations

Answer 8

The purpose of the reliability coefficient

• One time?
• Use repeatedly?

The nature of the test

• Is the construct homogeneous in nature? (life satisfaction)
• Or, heterogeneous in nature? (daily stress measure)

Question 9

Standard Error of Measurement

Answer 9

Used to estimate the extent to which an observed score deviates from the true score

Question 10

SEM

Answer 10

• SD of a theoretically normal distribution of test scores obtained by one person on equivalent tests
• The higher the standard error of measurement, the lower the reliability (inverse relationship w/ reliability)
• Provides a measure of precision of an observed score (allows us to estimate the range in which the true score is likely to exist)
• Related to confidence intervals

Question 11

Confidence Interval

Answer 11

• Range of test scores likely to contain the true score
• Observed score + or - (z critical value) * SEM

Ex: WAIS: You score was = 100; SEM = 2.5

Question 12

SEM & Confidence Intervals

Answer 12

• 90% confidence your true score is between 96.8 & 103.2
• 95% confidence your true score is between 95.1 & 104.9
• 99% confidence your true score is between 93.6 & 106.5

This is the case because 100 + or - the z score (1.28) x the SEM (2.5)

This equals 100 + or - 3.2z. You score is anywhere in the range of 100 + or - 3.2