Reliability Chapter 4 Flashcards
The Domain sampling model definition of reliability
Reliability is the relation of the variance of the observed score on a shorter test in the variance of the long run true score
Domain sampling model how is error introduced:
Problem and solution
By using a sample of items rather than the entire Domain of knowledge. The greater the items the higher the reliability
Classical test score Theory
Formula
Observed score equals true score plus error
Classical test Theory assumption
Error is random true score of a person doesn’t change with repeated applications of a test
Item response Theory explanation
Computer chooses a the questions to give an individual based on how many correct questions they have already got question difficulty goes up and down based on the individuals getting items of similar difficulty right or wrong
Item response Theory reliability
A shorter test has higher reliability
Sources of error
2 things.
Situational factors EG loud noises and temperature
Items not representative of the Domain
Three ways to estimate test reliability
Test-retest method
Parallel forms
Internal consistency
Time sampling the test retest method
Administer the test on two occasions correlate the scores
Problems with test-retest
2 things.
Carry over effects
practice effects
Maturation effects
Length of interval between test is important
Parallel forms methods
2 equivalent forms that measure the same atribute using different items
Sometimes given on the same day with counterbalanced order
One of the most rigorous assessments of reliability commonly in use
Parallel forms methods
2 equivalent forms that measure the same atribute using different items
Sometimes given on the same day with counterbalanced order
One of the most rigorous assessments of reliability commonly in use
Parallel forms methods
2 equivalent forms that measure the same atribute using different items
Sometimes given on the same day with counterbalanced order
One of the most rigorous assessments of reliability commonly in use
Split half method of reliability:
Definition
Problem
Solution
Two halves of a test
Correlatio
The spearman-brown formula corrects for using a shorter test
Estimates what the correlation between the two halves would have been if each half had been the length of the whole test
Internal consistency reliability split half how to overcome the problem that all halves are slightly different in difficulty
Kuder Richardson formula considers always of splitting a test in half
Split half reliability when to use cronbachs Alpha:
2 reasons
When variance is between the two halves is unequal
Scores have more options than right or wrong
Reliability of a difference score:
Definition and how to find it?
If you want to know the difference between performances at 2 points in time
Convert scores to Z units
Reliability for the difference score is lower than the reliability of the first and second measures
Reliability and behaviour observation studies:
Names for it
Best formula and why
Inter-rater reliability also called interscorer interobserver or into judge
Best method Kappa statistic
Kappa for chance agreement the servers are rating whether a behaviour occurred or did not occur there is a 50% chance of agreement
Standard errors of measurement and the rubber yardstick: problem and solution
The larger the standard error of measurement the less certainty about the accuracy with which the atribute was measured.
Researchers use confidence intervals to show the probability that the true score Falls within a certain interval around the observed score
How reLiable is reliable for basic research and for making decisions about people?
Basic research.70-.80
Instruments used to make important decisions .95
Increasing reliability of a test
Definition and formula
Increase the number of items. The prophecy formula can estimate how many more items a test needs to gain an acceptable level of reliability
Prophecty Formula assumptions the probability of error for new items = the probability of error for original items tiredness and boredom is not going to increase error
Discriminabilty analysis
Correlation between performance on single item and total test score if low item probably measures something different from the others.
Item may be too easy
Attenuation of correlation
Correlations are diminished by measurement error
Relibility needs Factor and item analysis because….
Tests are reliable if they are uni-dimensional one Factor should account for more variance than any other Factor
Correcting for attenuation
To Estimate what the correlation would have been between two measures without error
Find the reliability of both measures and the correlation between them
Use the correction for attenuation reliability formula
Use Krs20
Dichotomous
Use alpha
Likert skewed batiance
No right or wrong answer
Use pearson
Two continuous variables
