Reliability Flashcards
Reliability for CTT
For CTT; reliability was most investigated psychometric parameter & several statistical techniques were developed to estimate it
Corresponds to consistency of scores obtained by same individuals when they are re-examined with same test at different times, different sets of equivalent items or under other variable conditions of examination
Also concerns accuracy of measure or internal consistency (item-total correlation) between items of test
Clarifying concepts
Reliability= precision= accuracy
Nomenclature regarding stats techniques used for analysis of empirical data
Reliability; Internal consistency, equivalence, constancy & stability
Physical sciences->reliability->instrument calibration
Reliability= test ‘free from errors’
Measures of reliability
Correlation between 2 measures
Reliability coefficients= function of true variance (covariance) by total variance
Allows to estimate what proportion of total variance of test scores is error variance
> score= >reliability
measurement error
Error variance
Any condition that is irrelevant to objectives of a test represents error variance
Factors reducing error variance= standardisation of test administration, environmental control, instructions, true limits, normative sample
Test-retest reliability
Reliability coefficient is correlation between scores obtained by same individual in 2 different administrations of same test
Coefficient of stability or constancy
Ideal correlation is >0.6
Error variance source= interval between test administrations
Parallel (equivalent) forms
Reliability coefficient is correlation between scores obtained by same respondents in application or equivalent forms of test
Coefficient of equivalence
Ideal correlation is >0.7
True variance greater than error variance
Error variance source= content sampling
Split half
Reliability coefficient is correlation between scores obtained by same individuals in same test divided into 2 equivalent halves
Coefficient of internal consistency
Ideal correlation is >0.6
Error variance source= content sampling
Kuder-Richardson
Reliability coefficient is correlation between scores obtained in each item of test
Kuder-Richardson coefficient
Kuder-Richardson 20 formula used when items are dichotomous
Ideal coefficient is between 0.7-0.9
Error variance source= content sampling & content heterogeneity
Cronbach’s alpha
Reliability coefficient is correlation between scores obtained in each item of the test
Similar to Kuder-Richardson technique
Cronbach’s alpha formula is modification of Kuder-Richardson equation & reflects magnitude of covariance among items
Varies from 0-1 (1=100% internal consistency)
Ideal correlation is between 0.7-0.9
Error variance source= content sampling & content heterogeneity
Inter-rater reliability
Reliability coefficient is correlation between scores assigned by each rater
Error variance source= different raters perceptions
Factors affecting reliability
> sample variability, >reliability= >test accuracy
> number of items, >reliability= >test accuracy
Reliability for IRT
Reliability not as important as for CTT, though is part of validation process
Reliability can be obtained by estimating item information curve, which corresponds to graphical analysis that shows level of theta to which item brings maximum information
If amount of info small, ability can not be estimated with precision & estimates will be widely scattered about true ability
Item info function depends on item parameters a,b & x
Item information function
Can be used to predict scores of examinees at given ability levels
Test information curve
A result of the sum of the item info functions for all test items
A TIF value greater than or equal to 10.0 is considered ideal for ensuring good accuracy
Reliability for IRT
Reliability not as important as for CTT, though is part of validation process
Reliability can be obtained by estimating item information curve, which corresponds to graphical analysis that shows level of theta to which item brings maximum information
If amount of info small, ability can not be estimated with precision & estimates will be widely scattered about true ability
Item info function depends on item parameters a,b & x
