reliability pt 3

Question 1

inter rater reliability

Answer 1

-Applies when judgment must be exercised in scoring responses (e.g., WAIS-IV VCI subtests)

• Item level (agreement on item scores)
• How much agreement on each individual item?
• Correlation between raters on assigned item scores
• Scale level (total score on scale)
• How much agreement on the total score?
• Correlation between raters on total scores

If there are more than two raters, take the mean of the correlations for each pair of raters (A & B, B & C, A & C)

Question 2

inter rater reliability categorical decisions

Answer 2

When there is a finite number of categories to which each person being rated can be assigned

• Items: Pass/Fail (0,1)
• Items: 0, 1, 2
• Diagnosis: Present/Absent

Two methods for assessing:

• Percent Agreement
• Kappa

Question 3

inter rater reliability: percent agreement

Answer 3

• Percentage of all cases for which both raters make the same decision (i.e., both assign a score of 0 or both assign a score of 1)
• Problem: Raters could agree simply by chance
• Percent agreement can OVERESTIMATE inter-rater reliability
• Kappa (Κ) takes chance agreement into account and is the preferred method for assessing inter-rater reliability

Question 4

how to calculate percent agreement

Answer 4

Two raters independently decide whether an item score should be 0 or 1 for N individuals who complete the item
A = number of times item was scored 0 by both #1 and #2
B = number of times item was scored 0 by #1 and 1 by #2
C = number of times item was scored 1 by #1 and 0 by #2
D = number of times item was scored 0 by both #1 and #2
Percent agreement = percentage of cases for which both raters gave the same score (either both 0 or both 1) = (A+D)/N

Question 5

calculating chance agreement for a score = 0

Answer 5

Total scores of 0 given for Rater #1 = A + B
Total scores of 0 given for Rater #2 = A + C
Proportion of cases given a score of 0 by Rater #1 = (A+B)/N
Proportion of cases given a score of 0 by Rater #2 = (A+C)/N
Chance agreement for a score of 0 =
(A+B)/N times (A+C)/N

Question 6

calculating chance agreement for score = 1

Answer 6

Total scores of 1 given for Rater #1 = C + D
Total scores of 1 given for Rater #2 = B + D
Proportion of cases given a score of 1 by Rater #1 = (C+D)/N
Proportion of cases given a score of 1 by Rater #2 = (B+D)/N
Chance agreement for a score of 1 =
(C+D)/N times (B+D)/N

Question 7

calculating total chance agreement

Answer 7

Add the chance agreement for a score of 0 to the chance agreement for a score of 1
(A+B)/N times (A+C)/N
PLUS
(C+D)/N times (B+D)/N

Question 8

implications of reliability

Answer 8

• There is no single value that represents the reliability of a test … we must specify which type of reliability we are estimating
• The methods we have considered all permit us to estimate a specific type or source of error
• To estimate multiple sources of error simultaneously Generalizability Theory
• Test manuals will report all relevant types of reliability (test/retest; split-half; internal consistency; inter-rater)

Question 9

standard error of measurement

Answer 9

• Reliability coefficients apply to the test itself

- The SEM permits us to estimate how much error is likely to be present in an individual examinee’s score

Question 10

SEM in words

Answer 10

• Step 1. Subtract the reliability of the test from 1.
• Step 2. Take the square root of Step 1.
• Step 3. Multiply the standard deviation of the test by Step 2.

Question 11

SEM and reliability

Answer 11

• The SEM is INVERSELY -
• If reliability is high, SEM is low
• If reliability is low, SEM is high

Question 12

standard error of measurement according to classical reliability theory

Answer 12

According to Classical -Error is normally distributed around a mean of 0
-SEM = the standard deviation of the distribution of error scores

Using the probabilities associated with the normal curve

• The probability is 68% that the amount of error is within 1 SEM
• The probability is 95% that the amount of error is within 2 SEM

Question 13

estimating error

Answer 13

We can use the SEM to make probability statements about the amount of error associated with an observed score

NOTE: To do this accurately, we have to use the exact values rather than the “approximate” values we used in Chapter 1 of the Manual

• The probability is 68% that the amount of error associated with an observed score is no more than +/- 1 SEM
• The probability is 90% that the amount of error associated with an observed score is no more than +/- 1.65 times SEM.
• The probability is 95% that the amount of error associated with an observed score is no more than +/- 1.96 times SEM.

Question 14

confidence intervals for estimated true score

Answer 14

We can also construct confidence intervals around the estimated true score
-We can’t know the actual true score, but we can estimate it.

These confidence intervals tell us the range in which the person’s true score is likely to fall with a specified degree of certainty (probability)

These are the CI’s that are given in the table in the WAIS-IV Manual

Step 1. Calculate the estimated true score
Step 2. Calculate the standard error of estimate
Step 3. Calculate the desired confidence interval

Question 15

estimating the true score formula in words

Answer 15

Step 1. Subtract the Mean (M) from the observed score (Xo)
Step 2. Multiply Step 1 by the reliability of the test (rtt)
Step 3. Add the Mean to Step 2.

Question 16

standard error of estimate

Answer 16

Standard Error of Estimate (SEE) = SEM times reliability

Question 17

CI’s around estimated score….

Answer 17

…..will sometimes be asymmetrical around the obtained score

Reason: regression towards the mean

• The estimated true score will always be closer to the mean compared to the observed score
• Est True Score > Observed Score when observed score is below the mean
• Est True Score < Observed Score when observed score is above the mean

Question 18

difference between estimated true scores and observed scores

Answer 18

GREATER when

• Reliability is LOWER
• Observed Score is farther from Mean

LESS when
-Reliability is HIGHER
Observed Score is closer to Mean

Question 19

standard error of difference

Answer 19

• Used to decide if two scores are “significantly different” from one another
• i.e., the observed difference between them is NOT just due to measurement error

Question 20

how to find SED in words

Answer 20

• Step 1. Square the SEM of the first score
• Step 2. Square the SEM of the second score.
• Step 3. Add Steps 1 and 2
• Step 4. Take the square root of Step 3.

The SED will always be larger than the larger of the two SEMs

Question 21

using SED

Answer 21

• Multiplying the SED by 1.96 gives the amount of difference required for the scores to be considered significantly different at p < .05.
• For the VCI and PRI, this difference is 1.96 times 4.50, or 8.82
• The VCI and PRI must differ by at least 8.82 points (rounded to 9 points) in order for the difference to be considered statistically significant (not just due to measurement error) at p < .05.

Question 22

example of using SED

Answer 22

• In other words, differences less than 9 points could be due entirely to measurement error and therefore cannot be considered “true” differences
• VCI = 109 vs. PRI = 115. The difference is NOT statistically significant because the difference is only 6 points which is less than 9 pts.
• A difference that is less than 9 points could be due entirely to measurement error, i.e., the true scores actually might not differ from one another.

Question 23

SED and WAIS-IV

Answer 23

• To get more precise values for the minimum differences required for statistical significance at p < .05, we can use Table B.1 on p. 230 of the Administration Manual.
• This table calculates values using the reliability of the indices within each specific age range.
• Reliability of the indices varies slightly with age

For most purposes, the differences given in the WAIS-IV Interpretation Manual are sufficient