Reliability

Question 1

Reliability is based on…

Answer 1

the porportion of the total observed variance that is attributed to error

Total variation needs to have a certain amount of spread in order to get proper reliability.

Question 2

____ is crucial for reliabilty!

Answer 2

Variance

Wide range of scores is important

Question 3

Where does variability come from?

Answer 3

• Examiner
• Subject
• Instrumentation

Question 4

We want ____ for correlation and reliability

Answer 4

spread

Question 5

If subjects are homogenous it will result in…

Answer 5

the reliability coefficent to be low because the total variance is small

Question 6

Reliability formula

Answer 6

Observed Score = True Score ± error score

Question 7

Variance formula

Answer 7

s2 (score) = s2 (true) + s2 (error)

Question 8

A measurement is more reliable if a greater proportion of the total observed variance is _____

Answer 8

represented by the true score variance.

Question 9

What does a reliability of 1 mean?

Answer 9

Measurement has no error and therefore reflects the true score.

Question 10

Reliability Coefficent Range

Answer 10

• Range 0-1
• 0 = all meausre due to error
• 1 = true score, no error

Question 11

Why not use a correlation coefficent?

Answer 11

• Correlation do not reflect similarity of repeated scores. Correlations reflect covariance (rank order characteristics within the data) thus not consistency of ranks. Ex: Photo: two different lines but the correlations will be the same because of linear analysis.
• Product moment correlation designed to assess bivariate relationships (only two can be assessed)

Question 12

Advantages of an ICC

Answer 12

• Can be used to assess reliability of >2 ratings.
• Does not require the same number of raters for each subject (however, better to have all data). Leave box blank if you do not have data.
• Can be used on ordinal data.
• Supports generalizability theory (different testing occasions, characteristics of the raters/subjects, testing conditions and others).

Question 13

What model and settings do we use to analyze ICC on SPSS?

Answer 13

Model 2

Two-way random effects model - Absolute Agreement

Question 14

When show I use single vs mean scores for an ICC?

Answer 14

• Single: Most common, for individual raters or ratings are used. Intraclass coefficent
• Mean: measurments are unstable for several raters or ratings. interclass coefficent

3 Data trials on 2 different days
EX: Single: Take the 3 trials on one day and do an ICC
Ex: Mean: Get mean for each day then do an ICC.

Question 15

When examining a SPSS ICC Model 2 chart what do we need to look at specifically?

Answer 15

Only the Intraclass coefficent for the measure you analyzed. Single or Mean measures.

Question 16

An ICC of ____ is needed for most clinical measures

Answer 16

0.9 and above

Question 17

What makes an ICC low?

Answer 17

• Raters or ratings do not agree
• Too little variance, sample is too homogenous. Must have a heterogenous sample to avoid this!

Question 18

What is standard error of measurement?

Answer 18

• Referred to as typical error for a test
• Expected trial to trial difference swhen the true score does not change

Question 19

The SEM quantifies the ____ on a test

Answer 19

percision of individual scores

Question 20

Standard Error of Measurement (SEM) is not the same as ____

Answer 20

Standard error of mean (measure of vairability not necessarily percision)

Question 21

SEM is a function of the ____ value and ____ of the population

Answer 21

• ICC
• variability

Question 22

The greater the ICC value the smaller the ____ values. Why is this important?

Answer 22

• SEM

The smaller the SEM, the greater the sensitivity of your instrument to detect change. This is the responsiveness of the instrument, the more responsive the better.

Question 23

Minimum Detectable Change (MDC)

Answer 23

• Smaller MDC = greater responsiveness = good
• Tells you meaningful change. If I am above the value then real change has occured.
• MDC = 2.77 * SEM

Question 24

____ reliabe instrument -> ____ measurement error -> ____ MDC -> ____ responsiveness -> ____ capable of detecting change (good)

Answer 24

• Highly
• Low
• Small
• High
• More

Question 25

What is a kappa statistic?

Answer 25

• Use for nominal or oridinal data (Prefer Ordinal in an ICC)
• Range: 0-1 (Close to one is better, >0.8 is good for clinical practice)
• The more in the middle the more agreement for measurements.