W9 - Reliability & Validity Flashcards
Define reliability
Consistency of measurements, an ind’s performance on a test or the absence of measurement error.
Classical Test Theory
Spearman 1904
O = T + e
O = Observed score T = True score e = Error
What are the 2 types of error measurement?
Systematic error
Random error
Define systematic error
Consistent error which biases the true score + doesn’t affect reliability
Define random error
Unpredictable error which biases the true score + does affect reliability
Ways to minimise error
Train researcher to ensure proficient use of instrument
Repeats
Compare data from 2+ researchers
Careful design of study protocol
Consider choice of instrument
Calibrate instrument
Common technique used to assess relative reliability across time/researchers/writers…
Pearsons correlation coefficient
Higher correlation = ⬆️ reliability
How can relative reliability be assessed?
Through the test-retest reliability
= Assess the stability of the measurements on different occasions.
What is used when doing the test-retest reliability?
2 tests: Pearsons correlation coefficient
2 or + tests: Intraclass correlation coefficient (ICC)
Inter-rater reliability
Reliability / consistency across raters
Correlating the scores obtained from a group of participants by 2 or + researchers
What does internal consistency stand for?
Reliability across different parts of a measurement instrument
i.e items within a sub-scale on a questionnaire
How is internal consistency assessed?
Using Chronbach’s alpha reliability coefficient
Values range from 0-1
Closer to 1 = higher reliability
List some terms used for absolute reliability
Also known as measures of absolute reliability
Technical error of measurement
SE of measurement
Coefficient of variation
Limits of agreement
Define validity
Extent to which a test/instrument measures what its supposed to measure.
What are the types of validity?
Validity of measurement
Validity of a study
What comes under validity of measurement?
Face validity
Content
Construct validity
Criterion
Validity of measurement
What comes under criterion validity?
Concurrent
Predictive
What comes under validity of a study?
Internal
External
Define face validity
Whether the method of data collection obviously involves the factor being measured.
Define content validity
If the instrument adequately covers the domain of interest.
Define construct validity
Assesses extent to which an instrument accurately measures hypothetical constructs
What are the ways of assessing construct validity?
Convergent validity
Discriminant validity
ASSESSING CONSTRUCT VALIDITY
Convergent validity
Scores on an instrument to those on a similar measure
ASSESSING CONSTRUCT VALIDITY
Discriminant validity
Scores on an instrument are NOT related to those from an instrument which assess a different construct.
Criterion-related validity
Looks at whether the scores on an instrument are related to scores on a previously validated measure.
What are the ways of criterion-related validity?
Concurrent validity
Predicative validity
CRITERION-RELATED VALIDITY
Concurrent validity
Scores collected at roughly the same time
CRITERION-RELATED VALIDITY
Predicative validity
Criterion instrument completes at a later date
Commonly used technique to assess criterion-related + construct validity
Pearsons Correlation Coefficient
Can an instrument be reliable but not valid?
Yes
As it could be consistently measuring the wrong thing
Can an instrument be valid but not reliable?
NO
Internal validity
Refers to the ability to attribute changes in the dependent variable to the manipulation of the independent variable
External validity
Refers to the ability to generalise the results of a study to other settings + other individuals
Threats to internal validity
Maturation (Age/Growth) ?
Selection bias ?
Expecting certain results ?
Measurement + equipment? - can be overcome by freq calibration
Mortality ? (w/drawal / drop out)
Threats to internal validity
How can expecting certain results be avoided?
Blinding / double blind study
Threats to external validity
Reactive or interactive effects of testing
Interaction of selection of bias + treatment
Reactive effects of experimental arrangements
Multiple-treatment interference
THREATS TO EXTERNAL VALIDITY
How does reactive or interactive effects of testing have an influence?
Pre-test makes a participant more aware or sensitive to the treatment
THREATS TO EXTERNAL VALIDITY
How does interaction of selection bias + treatment have an influence?
Treatment is only effective in the group selected
THREATS TO EXTERNAL VALIDITY
How does Reactive effects of experimental arrangements have an influence?
Treatments effective in lab may not transfer to the real world
THREATS TO EXTERNAL VALIDITY
How does Multiple-treatment interference have an influence?
Effects of a previous treatment may influence subsequent ones
What is the definition of relative reliability?
The degree to which data maintain their position in a sample with repeated measurements
What is the definition of absolute reliability?
The degree to which repeated measurements vary for individuals
- Which of the following describes test-retest reliability?
a. Consistency across items
b. Consistency across raters
c. Consistency across time points
d. None of the above
Consistency across time points
When scores on an instrument are related to scores on a previously validated measure, which type of validity has been established?
Criterion-related validity
