Ch 3 - Reliability Flashcards
reliability =
the ability of test scores to be interpreted in a consistent and dependable manner across multiple test administrations
the property of reliability resides with what?
the test scores, not the test itself
if test reliability is high, how much weight do you give that test?
more weight
Classical test theory =
describes a set of psychometric procedures that can be used to test the reliability, difficult, and discriminatory properties of test items and scale
Classical Test Theory (CTT) equation:
X = T + E
raw score = true score + random error
*can try to reduce random error of tests
When error is minimized, a test produces ___?
more reliable scores
two sources of measurement error
systematic error - when a test is being used and consistently measures something other than the trait it was designed to assess (imperfect construct validity)
unsystematic error (random error) - collection of factors that contribute to the variation in scores across administrations - can be related to test, its construct, administration or scoring or to the individual
4 sources of measurement error =
- time sampling error
- content sampling error
- test administration error
- test taker variables
Time sampling errors result from ___?
repeated administrations of a test to the same individual
3 effects that can create a time sampling error =
- carryover effect - when an experimental treatment continues to affect a participant long after the treatment is administered
- practice effect - when individuals improve their scores across test administrations as a result of increased familiarity and comfort with a test and the content that is being assessed
- fatigue - when clients tire from multiple administrations of a test
why do counselors care about reliability?
if you know how reliable a test is, you are better prepared to determine what weight to assign the results of a test when making decisions
Correlation =
statistical technique used to measure and describe a relationship between two variables
*doesn’t speak to causation
positive relationship for correlation =
the scores on each of the two variables move in the same direction
negative relationship for correlation =
the scores move in opposite directions
correlation coefficient =
what is the range
numeric value that indicates the strength of the relationship between two variables
range between -1.00 and +1.00
what is a perfect correlation?
coefficient of -1.00 and +1.00 are perfect correlations
- means that each and every change in one variable is met with a constant change in the other variable
- the closer a number is to 1 in either direction, the stronger the correlation is
what does coefficient of 0 mean?
indicates no discernible relationship between the variables
what does positive/negative describe for a coefficient?
positive relationship means scores of the two variables move in the same direction (more study, better score)
negative relationship means the scores move in opposite directions (more depressive symptoms, lower the mood score)
What are the four methods of estimating reliability?
Test-retest reliability
Alternate forms reliability
Internal consistency
Inter-rater reliability
test-retest reliability =
assessing how reliable or stable scores on an instrument are over time - participants tested on same test on two separate times
3 challenges of test-retest reliability
carryover effect, maturation, memory loss
alternate forms reliability =
assessing stability using different but equivalent versions of a test
internal consistency reliability =
used to determine whether errors associated with content sampling are present (are the items consistent and assessing the same concept)
Inter-rater reliability =
used to assess level of agreement between two or more raters in their evaluation of a particular outcome (to what extent are the raters the same in how they rate a test)
Evaluating reliability coefficients table =
5 levels
very high = greater than 0.90 high = between 0.80 and 0.89 acceptable = between 0.70 and 0.79 Questionable = between 0.60 and 0.69 Unacceptable = less than 0.59
4 factors affecting reliability (significant for counseling)
- increase length of the test
- make sure test used is designed for population being assessed
- increase heterogeneity of group used to norm the test (was there diversity)
- Use optimal time interval between test administrations
