Reliability & Validity 1

Question 1

reliability

Answer 1

degree to which the same event produces the same result

-consistent, repeatable, dependable, reproducable

many reliabllity (agreement) estimates are based on measures of correlation (assoc), while also accounting for measurement error

Question 2

measurement error

Answer 2

“the noise”

Inc ME = des reliability

dec ME= inc reliability

Question 3

types of measurement error

Answer 3

systematic errors- predictable, consistent, and constant

random errors- unpredictable, inconsistent, variable

**great graph in notes on decription

Question 4

measurement error sources

Answer 4

examiner, examined, examination

Question 5

examiner-

Answer 5

systematic errors- consistent use of improper landmarks

random errors- fatigue, inattention

biological variation in senses, rely on sight, hearing an touch. Variation in acuity affects agreement (between clinicians, within the same clinician)

intratester, intertester, and test retest reliability

Question 6

examined

Answer 6

biological variation in the system

clinical attributes vary- HR, BP, pain intensity. these variations lead to inconsistencies.

Question 7

examination

Answer 7

disruptive environments. Affecting senses (dim lighting, noise environment). Privacy of setting.

Disruptive interactions.

Incorrect use of diagnostic tools.

Question 8

How to minimize measurement error

Answer 8

operational definition

proper training

inspection of equipment

indep interpretation of test results

“blinding” of examiner to diagnosis

seperate observaion from inference

Question 9

test retest reliability

Answer 9

measures the consistency of the testing instrument over time Most basic way to assess.

Question 10

internal consistency

Answer 10

measures the extent to which an instrument measures aspects of a certain characteristic (chronbach’s apha).

Question 11

intra-rater reliability

Answer 11

measures the stability of measures recorded by one individual across 2 or more trials

Question 12

inter-rater reliability

Answer 12

variation between 2 or more raters measuring the same group of subjects

Question 13

ICC

Answer 13

reflects both relationship and agreement

Cant measure reliability with correlation

compares between individual error and error variability (ratio)

ratio approaching 0 indicates no agreement

ratio approaching 1.0 suggests perfect agreement

no universal standards for interpretation

Question 14

ICC interpretation

Answer 14

> 0.90 = excellent reliability

> 0.75= good reliability

<0.75= poor to moderate reliability

Question 15

Pearson product moment correlation

Answer 15

major flaws as in its independent use as an indicator of reliability

why? it is a measure of association between ratings and not a true measure of agreement

Does not distinguish between perfect agreement and systematic bias

Question 16

quantifying reliability-continuous data

Answer 16

ICC appears to be best realiability coefficient option when analyzing cont data

Question 17

quantifying reliability-categorical variables

Answer 17

percent agreement

kappa coefficient (K statistic). chance corrected proportion of agreement.

kappa appears to be the best reliability coefficient option when analyzing categorical data

Question 18

kappa values

Answer 18

k=proportion of observed agreements- chance agreement/ 1-chance agreement

kappa=1.0 (perfect agreement)

kappa=0.0 (no better than chance agreement)

kappa=<0.0 (less than chance agreement)

**graph in notes

Question 19

bland-altman plots

Answer 19

another approach to agreement

plots extensively used to evaluate the agreement among two different instruments or two measurements techniques