Experimental Validity Flashcards
The goal of an experiment is to maximize ______ while maintaining _______
internal validity (eg, controlling extraneous variables, environmental variables)
external validity (difficulty generalizing to the real world)
What is a bias?
A systematic & non-random error that results in an incorrect (invalid) estimate of the association between the exposure and the outcome
True or False: Bias can be introduced by the researcher or study design
True
True or False: Bias cannot be corrected during analysis, only recognized and evaluated
True
What is bias away from the null and bias towards the null?
Bias away from the null
- Bias that augments the magnitude of a difference or association
- Leads researchers to conclude an association/difference seen in a sample exists in the population –> Type I error
Bias towards the null
- Bias that reduces the magnitude of a difference or association
- Leads researchers to conclude an association/difference observed in a sample occurred due to chance and does not exist in the population –> Type II error
What are 2 other sources of error that can lead to type I and type II errors not due to bias
Random Error
- Deviation from true relationship due to chance
- Cannot be attributed to a systematic flaw in research design or methodology
- Typically favors a Type II error
Confounding
- The effect observed in a sample is reflective of the population –> HOWEVER, the nature or cause of the effect is misleading
- Failure to account for or describe another variable that mediates the difference/relationship
What is sample bias and what type of validity is it a threat too?
Selection so that the sample that over- or under- represent certain characteristics of the population as a whole
Threat to: External Validity
- Your sample findings will be true for those with traits in common with sample –> might not be true for others with slightly different traits
What are the two types of sampling techniques and what re the strengths and weaknesses of both?
Probability Samples (“Random Selection”)
Strength: Sample is likely more representative of population (ie, Those with common traits in the population will be recruited more often, those with less common traits are recruited less often)
Weakness: While prevents bias, increases risk that sample may be due to population by chance (i.e. drawing 100 of 1000 may miss rarer traits)
Non-Probability Samples
Strength: Usually more feasible (ie, convenience)
Weakness: Some individuals are more likely to be recruited than others leading to a biased sample
What is selection bias, and what validity is it a threat too?
Systematic error in choosing participants to groups that alters associations between independent and dependent variable
Threat to: Internal Validity
- Clustering of individuals with a particular trait can lead to overestimation or underestimation of the true population
What is recall bias and what validity is it a threat too?
Inaccuracy in recalling exposure status, which results in differential misclassification
Threat to: Internal validity
- Differences may occur due to differences in recall of exposure rather than true differences
What is detection bias and what validity is it a threat too?
Bias due to different outcome assessments between groups
- Placebo Effect: Participation in a study, in and of itself, elicits an effect upon the dependent variable
- Halo Effect: Researcher’s a priori expectations of the effect of the independent variable biases dependent variable
Threat to: Internal validity
- Differences by groups may occur because of differences in assessment rather than due to exposure
What is instrumentation and how is it used in partner with measurement validity?
Instrumentation
- An instrument used to measure a variable should be reliable and valid
- *Measurement Validity**
- The extent to which an instrument measures what it is intended to measure
- For example:
- EEG or fMRI as a measure of brain activity or EMG as a measure of muscle function (construct validity)
- Does a sideline concussion test actually differentiate a concussed from non-concussed athlete (criterion validity)
- Does an exam accurately reflect knowledge of course material (content validity)
True or False: If a measurement is valid then it is reliable, but a measurement may be reliable without being valid.
True,(ie, shooting targets)
In addition to a measurement being Reliable and Valid we also want a measurement to be _____
Sensitive
- The probability that a measurement can detect changes in what it is supposed to be measuring
- number of true outcomes/ number or true + false outcomes
The measurement may be produce consistent results and may reflect the construct/criterion of interest but may not be sensitive to changes that delineate differential outcomes
What are the two types of maturation in an experiment and what validity are they a threat too?
Maturation
- An effect driven by natural change in the participants over time that is independent of an independent variable/intervention (ie, age development)
Selection Maturation
- An effect driven by natural change in the specific “characteristic” under study over time (ie, progression of disease)
Both threat: Internal validity
- Differences between groups may be due to different rates of maturation
