Session 7 - Reliability

Question 1

What is the most common mechanism used to establish reliability?

Answer 1

Test-Retest.

tests interest, intratester, intertester

Question 2

What measures of reliability are based upon parametric assumptions?

Answer 2

r, rsquared, and ICC or R.

Question 3

Parametric Assumptions

Answer 3

Between Group Variance
Within Group Variance
Sample Size
Number of Groups

Question 4

what does r=1 mean?

Answer 4

perfect correlation

Question 5

Pearson Product Moment Correlation Coefficient

Answer 5

Most common correlation coeficient
Can be used for reliability (test-retest, inter and intratester)
Can be used to assess strength of an association between 2 DIFFERENT variables with similar or different units of measure.

Question 6

Problems with Pearson’s (r)

Answer 6

Can only compare 2 groups
Measure of ASSOCIATION not CONCORDANCE–>
can vary directly, but be consistently different.
Validity is a weakness

Question 7

Coefficient of Determination (r-squared)

Answer 7

Percentage of variance accounted for when prediction one measurement from another.
Less useful than the SEM for judging the accuracy of a single value or set of values.

Question 8

Intraclass Coefficient Coreelation (ICC or R)

Answer 8

takes into consideration both association AND agreement
based upon a repeated measures ANOVA
Opposite of what we saw previously; BGV is the error rather than WGV b/c we want groups to be the same, (and overlap), rather than different.

Question 9

What is the difference between two curves with ICC?

Answer 9

tester error.

we want the curves to be identical

Question 10

Like Pearson, ICC is a parametric statistic, so normality of data is assumed and the statistics are sensitive to–

Answer 10

Between Group Variance
Within Group Variance
Sample Size
Number of Groups

Question 11

In what situation will you get a lower ICC/reliability coefficient when the actual magnitude of error is the same?

Answer 11

when tested it a homogenous population

Question 12

is statistical significance equal to clinical relevance?

Answer 12

NO!!

Question 13

how do you quantify magnitudes of error?

Answer 13

Coefficient of determination
Standard Error of Measurement
Minimal Detectable Change

Question 14

(SEM) Standard Error of Measurement

Answer 14

in the context of reliability, the SEM represents the variation in a measurement that is due to ERROR ALONE.
How much change in a measure is likely to be the result of random variability rather than “true change”

Question 15

(MDC) minimal detectable change or difference (MDD)

Answer 15

some identify the SEM as a measure of the error of any ONE measurement while the MDC is a quantification of the error introduced with repeated measurements.
the error of one measurement is compounded by having that much error again when the second measure is taken.

Question 16

(RCI) Reliability Change Index

Answer 16

RCI=(current measurement value-previous measurement value)/SD difference
RCI gives the probability that the measured change in a patient would have happened if the patient had not truly changed & the difference is, in fact, measurement error.

Question 17

what types of data do not meet parametric assumptions?

Answer 17

categorical, ordinal

Question 18

Cohen’s Kappa (K)

Answer 18

coefficient used for dichotomous or categorical data.
Agreement between judges adjusted for chance agreement.
Considered a true measure of concordance.

Question 19

how is a cohen’s kappa data table set up?

Answer 19

2x2 contingency table

Question 20

how is a weighted kappa set up? (Kw)

Answer 20

3x3 contingency table (greater than 2x2)
when there are more options that raters simply agreeing or not, it is possible to weight the seriousness or magnitude of disagreement.