Ch 5.1 Reliability

Question 1

Concept of Reliability

Answer 1

Consistency

• Within the test (Internal Consistency)
• Across different points in time (Test-Retest)
• Across different raters (Inter-rater) [Consistency across clinicians]

Question 2

All tests contain error

Answer 2

• Your observed test score is your “true” (T) score + some error (E)
• X = T+E

Question 3

Reliability

Answer 3

Repeatability or consistency of measurement

Question 4

Reliability coefficient

Answer 4

• Index of reliability
• Indicates ratio between true score variance & total variance on a test

Question 5

Variance

Answer 5

• Standard deviation squared
• Total variance = True variance (variance from true sources) + Error variance (variance from different sources)

Question 6

Smaller error variance

Answer 6

Small error variance = higher reliability coefficient (closer to 1)

Question 7

Larger error variance

Answer 7

Larger error variance = lower reliability coefficient (closer to 0)

Question 8

Measurement Error

Answer 8

• All factors associated with the process of measuring some variable, other than the variable being measured

Question 9

Random error

Answer 9

• Noise
• Unpredictable, inconsistent

Question 10

Systematic error

Answer 10

• Does not affect score consistency (reliability)
• However, does affect validity

Question 11

Sources of Error Variance [place holder]

Answer 11

.

Question 12

Test construction

Answer 12

• Item sampling
• Content sampling

Question 13

Test administration

Answer 13

• Environmental factors
  ex: noise level, lighting
• Test taker variables
  ex: emotional or medical state
• Examiner- related variables
  ex: I give you clues by the way I answer your questions

Question 14

Test scoring & interpretation

Answer 14

• Unstructured personality tests
• Manually scored exams

Question 15

Consistency Within the Test: Internal Consistency Reliability

Answer 15

• Use of a single test administered on one occasion to estimate reliability. Useful in understanding inter-item consistency (homogeneity)

Question 16

Split-half reliability

Answer 16

• Correlating scores obtained from equivalent halves
• Ways to split items
• Random split
• Odd-even split
• 1st & 2nd half
• Cannot simply correlate the 2 because shorter tests are generally less reliable than longer tests
• Must use statistics

Question 17

Split - Half Statistics

Answer 17

• When using split half reliability you use the Spearman Brown formula to calculate the full measures reliability coefficient

Question 18

Spearman Brown Formula

Answer 18

• Allows estimates of internal consistency from a correlation of two test halves (predicts reliability after changing test length)
• Calculate the effect of increasing or decreasing test length
• How many items needed to reach certain reliability

Question 19

Kuder Richardson Formula 20 (KR-20)

Answer 19

• Similar concept to Spearman-Brown
• Used when items are dichotomous right/wrong

Question 20

Cronbach’s Coefficient Alpha

Answer 20

• The mean of all possible split half correlations
• Used with continuous variable items
• Preferred statistics for obtaining an estimate of internal consistency reliability
• Range from 0 (no similarity) to 1 (perfectly identical)
• Larger the better but also you don’t want redundancy

Question 21

Consistency Between Tests: Parallel/Alternate Forms

Answer 21

• Degree of relationship between various forms of a test (equivalence)

Question 22

Parallel

Answer 22

• Must have the same means & variances of observed scores
• Estimate of the extent to which item sampling and other errors affect test scores

Question 23

Alternate

Answer 23

• Is equivalent with respect to content, difficulty level
• Time consuming and expensive to construct
• Advantages (controls memory effects)