4: Validity Reliability Flashcards
What contributes to soundness of experiments?
precision
accuracy
sensitivity
specificity
reliability
validity
what is precision?
Consistency, reliability, homogeneity of the data
what is accuracy?
Based on the precision of the measurements. Good accuracy if averaged values ≈ standard. Standards are rare in the biological sciences (behavioural or neurosciences). In psychophysics (Signal Detection Theory), accuracy = specificity + sensitivity
what is sensitivity?
Measures should be sensitive enough to detect differences in a characteristic that are important to the investigator
what is specificity?
Measures should be specific to the characteristics, group, phenomenon, etc. investigated
what is reliability?
Consistency of the measures. Precision
what is validity?
Does a variable represent what it is intended?
what are nuisance variables?
We hinted at this before with the concept of “distortion”
Two types:
Systematic error or bias
Random error (or error variance)
what is influenced by Systematic error or bias?
accuracy
what type of variables are a source of bias?
extraneous/ confounded variables
Types of biases/systematic errors and their solutions
Observer/Experimenter»>
Blinding procedures
Subject/Participant»_space;> Blinding/ Unobtrusive measures
Apparatus»_space;> Calibration
what is random error?
The precision of measurements (and therefore the consistency and reliability) is influenced by random error unpredictable fluctuations)
sources of random error
Random error is due to random fluctuations in participants, experimental conditions, methods of measurement, etc.
Main sources of random error:
*Observer / experimenter reliability
*Participant / subject reliability
*Instrument / apparatus reliability
contributors of precision
Calibration of an apparatus
Consistency of a participant
Environmental and other factors
Archery example: Bow/sight, archer, wind
How to assess precision?
Measures of variability (descriptive statistics)
measures of concordance
what are measures of variability
Standard deviation (sd) of repeated measurements.
Coefficient of variation (cv): (sd ÷ mean) × 100
what are measures of concordance?
Correlation coefficient: Consistency of results of paired
measurements. The coefficient of correlation is an index of
concordance
what is reliability?
Consistent results over repeated measurements. Reliability refers to the PRECISION of your measures
assessment methods
Test-retest reliability/consistency: Stability of test scores over time.
Alternative (parallel) forms reliability/consistency: e.g., recognition/recall example with
tests.
Internal consistency: How consistent is the measure across items intended to measure the same concept, e.g., split-half reliability/consistency » use of two lists in a memory test.
Inter-rater reliability: see next slide.
In some cases: Intra-rater reliability
what is Inter-observer/rater consistency or reliability?
consistency of recording
and scoring between ALL OBSERVERS
with an inter- observer reliability measure such as an index of concordance, kappa coefficient, Kendall coefficient
what is Intra-observer/rater consistency or reliability?
each observer, individually, records,
interprets or identifies SIMILAR behaviours or events the SAME
WAY.
with an intra- observer reliability measure)
what does intra mean
within
what does inter mean
between
what is validity about?
Validity is about the threats to valid inference making; Is the procedure you chose measures what it is intended to measure?
what are the four main types of threats to validity
construct validity
statistical conclusion validity
internal validity
external validity
what is construct validity
The wrong independent variables are identified
what is statistical conclusion validity
random error and wrong selection of statistical tests, low power, violation of statistical assumptions, fishing, etc
types of validity of a measurement
face validity
content validity
construct validity
criterion validity
what is face validity
How well the test appears to measure what it is designed to measure. It is a plausible measure of the variable we want to estimate. Face value. Non-scientific. E.g.: Common sense definition of stress.
what is content validity
How adequately the measure addresses the representativeness of the measured event or phenomenon as a whole (i.e., represents the whole content). Expert opinion can determine this type of validity. E.g., you lack content validity if you want to measure stress, but you only take behavioural measures and no physiological measures (or vice versa)
what is construct validity
A measure of how well a test and operational definition assess some underlying (theoretical) construct or variable. Depends heavily on the operational definitions, e.g., “stress”. The measurement procedure and the variable it measures are in agreement. An assay of glucocorticoids suggesting high levels of cortisol is associated with high stressful situations
what is criterion validity
The ability of a measure to assess (or predict) an outcome or criterion. Performance measures
what are the subtypes of criterion-related validity
concurrent validity
convergent/divergent validity
discriminant validity
predictive
what is concurrent validity
A measure of how well an assay estimates a criterion/ performance in relation to another (concurrent) phenomenon or group of subjects at the same point in time. A new test or assay is validated as it concurs with an older, better established one.
what is convergent/ divergent validity
Two or more methods of measurement converge/diverge upon one another. Strong relationship between the scores are found. Can be established by correlation
what is discriminant validity
The methods of measurement diverge upon one another and the divergence is expected. A measure of stress should not be expected to be highly correlated with a measure/construct of empathy
what is predictive validity
A measure of how well an assay predicts a phenomenon on a time criterion: e.g., pre/post. Measures predicts future states.
what is the relationship between validity and reliability
A measure can have high reliability but not low validity.
A measure cannot be more valid than it is reliable
types of validity and what they mean
Internal validity
External validity
what is internal validity and what does that mean
Measures what it is supposed to?
Associated with the criteria for ultimate (analytic) experiments (i.e., fully “experimental”).
- No confounded variables
- Controlled variables… are controlled
- Appropriate control group(s)
- Random assignment (randomization)
- Random selection (sampling) ~ preferable, but rarely attained
what is external validity and what does that mean
Generalization potential or generalizability
Generalizability of the data!
* Species
* Environments
* Cultures
* Age groups
* Conditions, etc.
Determines the applications and implications of an experiment.
what is the criteria for external validity
population selection
operational definitions
parameter values
demand characteristics
what is population selection
Converging evidence (from different populations) and representativeness of the sample
what are operational definitions
Agreement on definitions. For example, “stress”
what are parameter values
The values you select for each variable in your experiment should be well defined. Applies to control variables and independent variables
what are demand characteristics
Cues in a research procedure that influence the behaviour of subjects are absent or minimized.
Have the potential to influence internal validity
what is ecological validity ( case of external validity)
Related to external validity (generalizes/applies well to other people, settings, conditions, etc.).
Are experiments done in the laboratory generalizable to the “real world”?
Not a central concern of neuroscience (in general). Technological limitations
and constraints.
what are mediator variables
Provides a causal link in a sequence between an IV and a DV. Answers the WHY?
what are moderator variables
Modulates the strength or direction of the relation between an IV and a DV. Answers the WHEN, and for WHOM or WHAT?
