STATS 8 RELIABILITY

Question 1

Fundamentally, what is test reliability?

Answer 1

The precision with which the test measures the attribute.

Question 2

What is the statistical definition of reliability?

Answer 2

Reliability of the test score is the proportion of variance in the observed score due to the true score

Question 3

How can reliability be estimated using multiple measurements?

Answer 3

Reliability can be estimated using two independent measures of the same attribute. Since errors are uncorrelated (assumption 3) the only covariance between the two measurements should be due to true variance

Question 4

Explain the logic of the test-retest method of estimating reliability?

Answer 4

Assuming the attribute doesn’t change over time, the degree of error is the same on both occasions, and the errors on the two occasions are independent, reliability of the test score can be estimated as the correlation between Test and Retest.

Question 5

What are the limitations of the test-retest method of estimating reliability?

Answer 5

It assumes stability of the attribute over time. This typically underestimates reliability because any change in true score will be seen as error.

Question 6

Explain the logic of the parallel forms method of estimating reliability?

Answer 6

Assuming the two forms measure the same attribute, have the same degree of error, and the errors on the two forms are independent, reliability of the test score can be estimated from the correlation between scores on form 1 and form 2.

Question 7

What are the limitations of the parallel forms method of estimating reliability?

Answer 7

Influenced by the precision with which the two forms measure the attribute, and the extent to which the two forms measure the same attribute. Reliability estimate may be distorted due to lack of parallelism

Question 8

Explain the logic of the split-half method of testing reliability?

Answer 8

Assuming the two halves contribute the same amount to the measurement of the attribute, have the same degree of error, and the errors of measurement are independent, reliability of score on either half can be estimated as the correlation between half 1 and 2

Question 9

What is the spearman-brown formula in split-half reliability?

Answer 9

Split-half reliability only tells us the reliability of each half, not the whole test, spearman-brown estimates the reliability of the whole test from the split-half reliability

Question 10

What are the limitations of the split-half method of testing reliability?

Answer 10

There are many ways the test can be split, and each split will give a different estimate of reliability

Question 11

Explain the logic of the internal consistency (alpha) method of testing reliability?

Answer 11

Split the test into items. Assuming each item contributes the same amount to the measurement of the attribute, and that errors contributed by items are all independent, reliability of an average item can be estimated as the average inter-item correlation.

Question 12

Why does the internal consistency method use Cronbach’s Alpha?

Answer 12

Alpha estimates the test reliability from its individual items. The logic is the same as the spearman-brown formula in the split-half method.

Question 13

What can we show that Alpha is?

Answer 13

Alpha is the mean of all possible split-half coefficients