Reliability Flashcards
What does reliability in a test mean?
Consistent results between administrations.
Is validity essential for reliability?
No. Reliability is essential for validity.
How are reliability and validity expressed? Also, what is the shorthand used in equations for both?
They are expressed as correlations.
Reliability = rxx
Validity = rxy
What two things does rxx stand for?
The correlation between scores (x) on two administrations of a test.
The proportion of obtained score variance explained by true score variance. (0.8=80% explained by true score variance).
What does true score theory state?
If a test is given to an individual an infinite number of times, their results would create a normal distribution and the mean would be their true score. The standard deviation would be the standard error of the estimate.
If rxx = 0.8 what amount of variance is due to true score variance and what amount is due to error score variance?
80% is due to true score variance and 20% is due to error score variance.
When will a persons raw score = their true score
When error = 0 or rxx = 1. There is a perfect correlation…
What two measures of variance do we need to add to get the total variance in obtained score
true score variance and error variance.
What are the five types of reliability ?
- Test-retest
- Alternate form (immediate)
- Alternate form (delayed)
- Internal consistency
a. split-half
b. Kuder Richardson and Cronbach’s alpha - Scorer (inter-rater)
What occurs in test-retest reliability?
Two tests are given to the same group of people on two different occasions.
Goldy locks zone is 4 weeks apart.
What type of correlation does test-retest give us and what type of error variance?
Coefficient of stability (how stable over two tests)
Time sampling
What are the two ways Alternate forms reliability is administered?
Two versions of test constructed in an identical way, except they contain different content. These are then administered to the same people, in one condition one after another, in another at separate times.
What type of error variance does alternate forms measure?
Immediate = Content sampling Delayed = Time and content sampling
How is Split-half administered and how is the correlation measured?
A single test given and then split into two equal halves.
The correlation is measured by comparing the scores on each half of the test.
What type of error varaince does split half produce?
Content sampling
What does Spearman-Brown formula allow us to do?
It lets us look at the effect of lengthening and shortening a test.
What is the difference between Kuder-Richardson and Cronbach’s alpha? Also, what type of error variance is measured?
KR is used for dichotomous answers, while Cron’s is used for Likert-scales.
These both look at content sampling and content heterogeneity.
What does scorer reliability measure?
Interscorer differences.
How many test forms and on how many occasions do the 5 reliability tests need to be given?
test-retest = one form, on two occasions
Alternate forms immediate = two forms, one occasion
Alternate forms delayed = two forms two occasions
Split half = one form, one occasion
KR-20 & Cron’s = one form, one occasion
Scorer = one form, one occasion
How high should reliability be for a good test?
Greater than 0.8
How do you estimate a confidence interval around a true score?
First find the true score ti = M+rxx(xi-M)
Then calculate the standard error of measurement
SEM= SD square root of 1-rxx
Then calculate CI’s
ti+/- 1.96 X SEM
What is the true score if the rxx equals 0 or 1
for 0, true score will equal the mean
for 1 true score will equal the raw score.
Should we consider the difference significant between scores if their percentile bands overlap?
No
How do we test if there is a significant difference between two scores?
We use the standard error of difference. This will calculate how large the difference needs to be for it to be significant. We then find the difference between the two scores and see if it is bigger than our SEdiff.
