5010: Reliability, Responsiveness

Question 1

Types of Measurement Error

Answer 1

Systematic
(consistent, unidirectional, biased, “constant”)

Random
(inconsistent, either direction equally likely, try to minimize, on average- will cancel out)

Question 2

Sources of Measurement Error

Answer 2

Rater (stabilization, recording)

Meas. Instrument or Method (goni-faulty, consistency- interrater)

Subject (clothing, m. mass, gender, time of day, meds)

Question 3

Types of Reliability

Answer 3

Intrarater (usu. MOST reliable)
Interrater
Test-retest (suggests no rater involvement, self-reported data)

Question 4

Intraclass Correlation Coefficient
(ICC)

Answer 4

True score variance b/w subjects
= ————————————————
Total variance

Question 5

Reliability Coefficient

Answer 5

True score variance b/w subjects
= ————————————————
Total variance + error variance

Question 6

Variance

Answer 6

Measure of avg. variability of sample data. Ideally for a clinical measure.

Question 7

Interpretation of Reliability Stats

Answer 7

Range: 0-1

Question 8

Statistical Measures of Reliability

Answer 8

ICC- for Continuous (sometimes Ordinal)
SEM for Continuous
Kappa for categorical
Cronbach’s alpha for Multiple items, one meas.

Question 9

Bland-Altman Plots

Answer 9

Plot difference between test-retest

Shows repeatability and any bias over time

Question 10

Validity

Answer 10

Are we really measuring what we think we are measuring?

Question 11

4 Categories of Validity testing

Answer 11

Face Validity
Content Validity
Criterion-related validity
Construct Validity

Question 12

Face Validity & How to Test

Answer 12

Does it appear to be valid for this measurement (subjective)

1. Have clinicians look it over & give opinion
2. Perform the test on patients & ask their opinion
3. Clinician or Pt may reject it

Question 13

Content Validity & How to Test

Answer 13

Does instrument address all aspects and only aspects of the attribute being measured?

1. 1st author must give definition of what they intend to meas.
2. Use a thorough, organized, comprehensive development process.
3. May sample expert opinion
4. Test for ceiling & floor effects
5. Analyze data using factor analysis

Question 14

What is Factor Analysis

Answer 14

Testing tool used to test CONTENT VALIDITY

• used for multi-item meas. tools
• complex statistical analysis based on correlation among items
• will identify # and type of underlying dimensions being meas
• may identify items that do not correlate/fit w/ other items

i.e: balance test –> might also identify strength –> but that is not what we are trying to measure.

Question 15

Criterion-Delated Validity & Test

Answer 15

this test requires a “gold standard” to serve as criterion.
Goniometry meas- gold standard is w/ x-ray

We don’t always use them because of cost, time, practicality, and the may be uncomfortable

2 Ways to Test:
1. Concurrent- gold standard used @ same time
2. Predictive- gold standard is some future event
(GRE to predict PT success)
To measure, calculate the correlation b/w the measurement and the gold standard.

Question 16

Construct Validity

Answer 16

Used for abstract attributes that are difficult to define, where there is NO “gold standard”
Construct= logical argument/hypothesis about how a given means should behave (if it is measuring what you think it is measuring)
Researcher constructs argument, then test that argument via hypothesis testing.
NEVER QUITE FULLY PROVEN

Question 17

Theoretical Models (Framework)

Answer 17

helps define a variable by stating its relationship to other variables and phenomena

provides a basis for construct validity testing

i.e.: personal/environmental factors
Fear-Avoidance Model of Pain

Question 18

Construct Validity: Convergent vs. Discriminant

Answer 18

Convergent: correlation w/ other established balance measures (positive or negative)

Discriminant: No correlation with what we are testing.
(might correlate w/ another aspect like “cognition”, but if we are testing balance, it is discriminant)

Question 19

Look at Self-Check on pg 168 of course pack

Answer 19

:)

Question 20

Responsiveness

Answer 20

Ability of an instrument to detect clinically important change over time.

Essential for outcome measure when you expect to see progress in response to your treatment

Question 21

Things to ask…

Answer 21

Time Frame?

Function Measure Has Adequate Responsiveness?

Question 22

How do researchers meas/compare responsiveness?

Answer 22

I. Must have intervention &/or pt pop. where change is expected.
II. Must follow puts over sufficient time to see change
III. Calc. responsiveness for a # of outcome measures and compare.

Baseline———->Discharge Meas.s & Global Change Meas.s

Question 23

Global Change Measure /

External Criterion

Answer 23

Responsiveness involves detecting change in subjects who have actually changed.

In an sample, many pts improve, but some stay same or get worse.

Authors may use global change meas. to determine change status. (Improved/Same or stable/Worse)

Question 24

GROC

Answer 24

Global Rating of Change

A transitional measure

Question 25

Other Transitional Measures

Answer 25

How have your symptoms changed since you began treatment?

i. complete recovery
ii. much improved
iii. slightly improved
iv. no change
v. slightly worse
vi. much worse
vii. worse than ever

Question 26

Responsiveness Statistics

&

Advantage

Answer 26

Effect Size
Standardized Response Mean (SRM)

Advantage: Both statistics are unit-less, making comparisons b/w outcome measures easier (even if scales are different)

Question 27

Effect Size

Answer 27

Mean Change
= ————————
Baseline of sd

   (Final-Initial) =    -----------------------
   Baseline of sd

Question 28

Standardized Response Mean (SRM)

Answer 28

Mean Change
= ——————————
sd of change scores

        (Final-Initial) =    ------------------------------
   sd of change scores

Question 29

Interpretation of Cohen’s Effect Sizes

Answer 29

Effect Size &/or SRM*

.8 Large
.5 Moderate
.2 Small

*Both normalized (in sd units)

Question 30

Population

Answer 30

• remember reliability, validity, and responsiveness are specific to a given population
• can’t assume it will work in a diff. context
• -> try to reference a study where population is similar to pt in your clinic care

Question 31

Practicality

Answer 31

Consider

• time
• expense
• setting

May depend on context of your measurement
(research study, clinical practice)