Reliability

Question 1

What does reliability in a test mean?

Answer 1

Consistent results between administrations.

Question 2

Is validity essential for reliability?

Answer 2

No. Reliability is essential for validity.

Question 3

How are reliability and validity expressed? Also, what is the shorthand used in equations for both?

Answer 3

They are expressed as correlations.
Reliability = rxx
Validity = rxy

Question 4

What two things does rxx stand for?

Answer 4

The correlation between scores (x) on two administrations of a test.
The proportion of obtained score variance explained by true score variance. (0.8=80% explained by true score variance).

Question 5

What does true score theory state?

Answer 5

If a test is given to an individual an infinite number of times, their results would create a normal distribution and the mean would be their true score. The standard deviation would be the standard error of the estimate.

Question 6

If rxx = 0.8 what amount of variance is due to true score variance and what amount is due to error score variance?

Answer 6

80% is due to true score variance and 20% is due to error score variance.

Question 7

When will a persons raw score = their true score

Answer 7

When error = 0 or rxx = 1. There is a perfect correlation…

Question 8

What two measures of variance do we need to add to get the total variance in obtained score

Answer 8

true score variance and error variance.

Question 9

What are the five types of reliability ?

Answer 9

1. Test-retest
2. Alternate form (immediate)
3. Alternate form (delayed)
4. Internal consistency
   a. split-half
   b. Kuder Richardson and Cronbach’s alpha
5. Scorer (inter-rater)

Question 10

What occurs in test-retest reliability?

Answer 10

Two tests are given to the same group of people on two different occasions.
Goldy locks zone is 4 weeks apart.

Question 11

What type of correlation does test-retest give us and what type of error variance?

Answer 11

Coefficient of stability (how stable over two tests)

Time sampling

Question 12

What are the two ways Alternate forms reliability is administered?

Answer 12

Two versions of test constructed in an identical way, except they contain different content. These are then administered to the same people, in one condition one after another, in another at separate times.

Question 13

What type of error variance does alternate forms measure?

Answer 13

Immediate = Content sampling
Delayed = Time and content sampling

Question 14

How is Split-half administered and how is the correlation measured?

Answer 14

A single test given and then split into two equal halves.

The correlation is measured by comparing the scores on each half of the test.

Question 15

What type of error varaince does split half produce?

Answer 15

Content sampling

Question 16

What does Spearman-Brown formula allow us to do?

Answer 16

It lets us look at the effect of lengthening and shortening a test.

Question 17

What is the difference between Kuder-Richardson and Cronbach’s alpha? Also, what type of error variance is measured?

Answer 17

KR is used for dichotomous answers, while Cron’s is used for Likert-scales.
These both look at content sampling and content heterogeneity.

Question 18

What does scorer reliability measure?

Answer 18

Interscorer differences.

Question 19

How many test forms and on how many occasions do the 5 reliability tests need to be given?

Answer 19

test-retest = one form, on two occasions
Alternate forms immediate = two forms, one occasion
Alternate forms delayed = two forms two occasions
Split half = one form, one occasion
KR-20 & Cron’s = one form, one occasion
Scorer = one form, one occasion

Question 20

How high should reliability be for a good test?

Answer 20

Greater than 0.8

Question 21

How do you estimate a confidence interval around a true score?

Answer 21

First find the true score ti = M+rxx(xi-M)
Then calculate the standard error of measurement
SEM= SD square root of 1-rxx
Then calculate CI’s
ti+/- 1.96 X SEM

Question 22

What is the true score if the rxx equals 0 or 1

Answer 22

for 0, true score will equal the mean

for 1 true score will equal the raw score.

Question 23

Should we consider the difference significant between scores if their percentile bands overlap?

Answer 23

No

Question 24

How do we test if there is a significant difference between two scores?

Answer 24

We use the standard error of difference. This will calculate how large the difference needs to be for it to be significant. We then find the difference between the two scores and see if it is bigger than our SEdiff.