Week 5: Measurement Reliability

Question 1

What is the minimum detectable change?

Answer 1

The smallest amount of change that can be detected by a measure that corresponds to a noticeable change in ability.

Question 2

What does the reliability of a study represent?

Answer 2

The stability of the measurement.

Question 3

What does the validity of a study represent?

Answer 3

The meaningfulness of the measurement.

Question 4

What are the four data types? Provide an example for each.

Answer 4

1. Continuous: timed up and go test
2. Ordinal: stroke severity i.e. mild, moderate, severe
3. Nominal: categorical i.e. AFL, soccer, rugby
4. Dichotomous: injured, not injured

Question 5

What are the different types of reliability?

Answer 5

1. Inter-rater: between different assessors
2. Intra-rater (test-retest): within one assessor
3. Internal consistency: agreement between items that measure the same construct e.g. questionnaire items that measure the same concept.

Question 6

How can reliability be measured?

Answer 6

• Kappa
• Bland-Altman plot
• Minimum detectable change
• Intra-class correlation

Question 7

How is Kappa used to measure reliability?

Answer 7

1. Use Kappa 2 x 2 table
2. Calculate the chance agreement
3. Calculate the proportion of agreement beyond that expected chance.
   K = observed agreement - chance agreement

Question 8

What do Kappa scores means?

Answer 8

• Close to +1 means high level of agreement

- Less than 0 means worse than flipping a coin.

Question 9

What are the other types of Kappa?

Answer 9

1. Weighted: gives points for partially correct answer

2. Prevalence adjusted, bias-adjusted kappa (PABAK) e.g. when yes:no split is 5:95% rather than 50:50.

Question 10

What does a Bland Altman plot involve?

Answer 10

• Plot the difference between the two measurement devices against the average of those two measurements.
• Calculate the mean of the differences. The average of the difference should ideally be close to 0.
• Calculate SD+1 and SD-1

Question 11

How is proportional bias interpreted from a Bland-Altman plot?

Answer 11

Negative number means second test is giving bigger measurement.
Positive number means first time is giving a bigger measurement.

Question 12

What does SEM stand for?

Answer 12

Standard error of measurement (SEM)

Question 13

What can be concluded if the difference between pre-treatment score and post-treatment score is bigger than the MDC?

Answer 13

Then we are 95% confident that true change has occurred.