Week 5 - Reliability and Validity Flashcards
internal and external validity are considered in a what?
study
reliability and validity are considered in a what?
measure
when evaluating a ___ discuss the internal and external validity
study
when evaluating a ___ discuss the reliability and validity.
measure
- process of assigning numerals to variables to represent quantities of characteristics according to certain rules.
- approach to detecting and documenting relative conditions or events.
measurement
____ decreases ambiguity and increases understanding via the expression of qualitative/quantitative info about a given variable.
measurement
numbers represent units with equal intervals, measured from true zero.
ratio scale
name 3 examples of ratio measurements.
distance, age, time
numbers have equal intervals but no true zero.
interval scale
name 2 examples of interval measurements.
calendar years, temperature
numbers indicate rank order
ordinal scale
name 2 examples of ordinal measurements.
mmt, pain
numerals are category labels.
nominal scale
name 2 examples of nominal measurements.
gender, blood type
some level of inconsistency is inevitable
measurement error
name 3 sources of inconsistency in measurements.
- tester (rater)
- instrument
- subject or character itself
describe the formula for observed score.
observed score (x) = true score(T) +- measurement error (E)
- consistent, unidirectional, and predictable (if detected).
- relatively easy to correct; recalibration or add or subject the correction.
- a concern of validity
systematic errors
occur by chance and alter scores in unpredictable ways; chance fluctuations (tend to cancel out over repeated measurements)
random errors
name 2 examples of systematic errors.
illiteracy, confusing terms
name 3 examples of random errors.
mood, level of fatigue, motivation
___ ___ are generally not influenced by magnitude of true score.
random error
the ____ the sample, the more the random errors are cancelled out.
larger
name 4 common sources of error.
- respondent
- situational factors
- measurer
- instrument
- not all error is random
- some error components can be attributed to other sources, such as rater or test occasion.
generalizability theory
the consistency of your measurement instrument
reliability
the degree to which an instrument measures the same way each time it is used under the same condition with the same subjects
reliability
reflects how consistent and free from error a measurement is (ex: reproducible/dependable)
reliability
reliability estimates are based upon score variance: the variability or distribution of scores
reliability coefficient
how is reliability/reliability coefficient measured? (formula)
reliability coefficient = true variance/ (true variance + error variance)
describe the range of the reliability coefficient.
<0.50 = poor
0.50-0.75 = moderate
>0.75 = good
(the closer to 1 the better)
reflects the degree of association or proportion between scores
correlation
reflects the actual equality of scores
agreement
do not affect reliability coefficient since relative scores remain consistent (high correlation).
systematic errors
name the 4 types of reliability.
- test-retest reliability
- rater reliability
- alternate forms reliability
- internal consistency
-indicates the stability (consistency) of an instrument through repeated trials.
test-retest reliability
addresses the rater’s influence on the accuracy of the measurement
intra-rater reliability
addresses the variation between separate raters on the same group of participants.
inter-rater reliability
how is test-retest reliability and rater reliability assessed?
intraclass correlation coefficient (ICC) or kappa
- equivalent or parallel forms reliability
- eliminates memory of particular responses in traditional test-retest format.
alternate forms reliability
- homogeneity; the degree of relatedness of individual items measuring the same thing (factor/dimension)
- how well items “hang together”
internal consistency
how is alternate forms reliability assessed?
correlation coefficients
how is internal consistency assessed?
cronbach’s coefficient alpha
reliability exists in a ____.
context
reliability is not ____. it exists to some extent in any instrument.
all-or-none
name 6 ways to maximize reliability.
- standardize measurement protocols
- train raters
- calibrate and improve the instrument
- take multiple measurements
- choose a sample with a range of scores
- pilot testing
how consistent it is given the same conditions
reliability
if it measures what it is supposed to and how accurate it is
validity
the degree to which an instrument actually measures what it is meant to measure
validity
how is validity determined?
by the relationship btwn test results and certain behaviors, characteristics or performances.
____ is a prerequisite for ____, but not vice-versa.
reliability, validity
name the 4 types of measurement validity.
- face validity
- content validity
- criterion-related validity
- construct validity
instrument appears to test what it is supposed to and it seems reasonable to implement; subjective process
face validity
what is the weakest form of validity?
face validity
instrument adequately addresses all aspects of a particular variable of interest and nothing else; subjective process by “panel of experts” during text development; non-statistical procedure
content validity
new instrument is compared to a “gold standard” measure; objective and practical test of validity
criterion-related validity
target and criterion measures are taken relatively at the same time
concurrent validity
target measure will be suitable predictor of future criterion score
predictive validity
name an example of predictive validity.
the SAT
- instrument effectively measures a specific abstract ideal.
- reliant upon content validity of construct and underlying theoretical context
construct validity
name 5 methods of construct validation.
- known groups method
- convergent and divergent validity
- factor analysis
- hypothesis testing
- criterion validation
two measures believed to reflect the same underlying phenomenon will yield similar results or will correlate highly.
convergent validity
indicates that different results or low correlations are expected from measures that are believed to assess different characteristics.
divergent validity
a test to discriminate btwn 2 or more groups.
discriminant validity
name the 2 main types of construct validity.
convergent and divergent validity
name the 2 main types of criterion-related validity.
concurrent and predictive validity
the ability of an instrument to accurately detect change when it has occurred.
responsiveness to change
smallest difference in a measured variable that subjects perceive as beneficial.
minimally clinically important difference (MCID)
a standardized assessment designed to compare and rank individuals within a defined population.
norm-referenced test
interpreted according to a fixed standard that represents an acceptable level of performance.
criterion-referenced test
name 3 things that change scores are used to do.
- demonstrate effectiveness of an intervention.
- track the course of a disorder over time.
- provide a context for clinical decision making
the smallest difference that signifies an important difference in a pts. condition
minimal clinically important difference (MCID)
more meaningful for the subjects and clinicians
clinically important data
the methods and measures we used for the study are good and will produce valid results.
internal validity
relates to how well we can generalize the findings of the study to the entire population we’re interested in.
external validity
must have ___ validity to also have ___ validity.
internal, external
way to conceptualize a variable to reduce ambiguity about it.
measurement
___ errors are harder to correct.
random
what is the first step in making a measure standardized?
reliability
administer a test twice to assess agreement btwn the 2 tests
test-retest reliability
participants could get better the second time they take the test.
practice effect
statistic that reflects both agreement and correlation
ICC (intraclass correlation coefficient)
one rater; assess the same person twice to see whether your scoring has changed
intra-rater reliability
considers the constructs rather than the consistency of the measurements.
factor analysis
