Chapter 9: Reliability

Question 1

What are the concepts of reliability?

Answer 1

Measurement error
Observed score = true score ± error component

Question 2

What are types of measurement error?

Answer 2

Systematic
Random

Question 3

What are sources of measurement error?

Answer 3

The person taking the measurements
The measuring instrument
Variability in the characteristic: ex - BP

Question 4

What is the generalizability theory?

Answer 4

Not all error is random
Some error components can be attributed to other sources, such as rater or test occasion.

Question 5

What is relative reliability?

Answer 5

Reflects true variance as a proportion of total variance in a set of scores
Measured as a unitless coefficient
Intraclass correlation coefficients (ICC) and kappa coefficients are commonly used

Question 6

What is absolute reliability?

Answer 6

Indicates how much of a measured value, expressed in the original units, is likely due to error
Standard error of the measurement (SEM) is commonly used

Question 7

What should we understand about reliability?

Answer 7

Reliability exists in a context
- Relevant to a tool’s application
Reliability is not all-or-none
- Exists to some extent in any instrument

Question 8

What are types of reliability?

Answer 8

Test-retest
Rater
Alternate forms
Internal consistency

Question 9

What are considerations for test-retest reliability?

Answer 9

Most meaningful for measures that do not rely on raters
The interval between tests
- To support stability of the measurement
Carryover
- From practice or learning
Testing effects
- Act of measurement changes the outcome

Question 10

What is intrarater reliability?

Answer 10

one rater

Question 11

What is interrater reliability?

Answer 11

2+ raters
Best when all raters measure the same response

Question 12

What are change scores?

Answer 12

With large error variance, the difference from trial 1 to trial 2 may cancel out true score and be composed of mostly error

Question 13

What is regression toward the mean?

Answer 13

Tendency for extreme scores to fall closer to the mean upon retesting

Question 14

What is minimal detectable change?

Answer 14

Based on the standard error of the measurement (SEM)
Amount of change that goes beyond error
Also known as minimal detectable difference, smallest real difference, smallest detectable change, coefficient of repeatability, or the reliability change index.

Question 15

How can we maximize reliability?

Answer 15

Standardize measurement protocols
Train raters
Calibrate and improve the instrument
Take multiple measurements
Choose a sample with a range of scores
- Must have variance in scores to show reliability
Pilot testing

Question 16

What is classical measurement theory?

Answer 16

any observed score (Xo) consists of two components: a true score (Xt) that is a fixed value, and an unknown error component (E) that may be large or small depending on the accuracy and precision of our measurement procedures.

Question 17

What is measurement error?

Answer 17

Any difference between the true value and the observed value

Question 18

What is systematic error?

Answer 18

Systematic errors are predictable errors of measurement. They occur in one direction, consistently overestimating or underestimating the true score.
Systematic errors are consistent. Consequently, systematic errors are not a threat to reliability. Instead, systematic errors only threaten the validity of a measure

Question 19

What is random error?

Answer 19

Presuming that an instrument is well calibrated and there is no other source of systematic bias, measurement errors can be considered “random” since their effect on the observed score is not predictable.
These errors are a matter of chance, possibly arising from factors such as examiner or subject inattention, instrument imprecision, or unanticipated environmental fluctuation.

Question 20

The intraclass correlation coefficient is used with what type of data?

Answer 20

quantitative

Question 21

The kappa coefficient is used with what type of data?

Answer 21

categorical

Question 21

What is alternate forms reliability?

Answer 21

Alternate forms reliability, also called equivalent or parallel forms, assesses the differences between scores on the different forms to determine whether they agree.

Question 22

What is internal consistency?

Answer 22

Internal consistency, or homogeneity, reflects the extent to which the items that comprise a multi-item test succeed in measuring the various aspects of the same characteristic and nothing else.