Validity Flashcards
Statistical conclusion validity
(were stats appropriate?)
Validity of inferences about the correlation (covariation) between treatment and outcome.
- Ensure adequate sample size for sufficient statistical power.
- Apply appropriate statistical techniques and correct for multiple testing.
- Control for random error and use reliable measurement tools
Internal validity
(did A cause B?)
Validity of inferences about whether observed covariation between A (presumed treatment) and B (the presumed outcome) reflects a causal relationship from A to B as those variables were manipulated or measured.
Randomization!!!
- Use randomization to assign participants to groups.
- Implement control groups and use blinding.
- Employ a pretest-posttest design.
- Ensure consistent treatment administration.
External validity
(can lessons be extended beyond the lab?)
Validity of inferences about whether the cause-effects relationship holds over variation in persons, settings, treatment variables, and measurement variables.
Generalizability!!!! and replicate!!!!
- Use random sampling to select participants.
- Replicate studies in different settings and with different populations.
- Test the intervention across various conditions.
Construct validity
(what is A?)
Validity of inferences about higher-order constructs that represent sample particulars.
- Clearly define and operationalize constructs.
- Use multiple methods and measures to assess constructs.
- Validate the measurement tools with pilot studies.
Ecological validity
(does the lab context mirror the real world?):
Ambiguous temporal precedence
Internal validity
A causes B? or vice versa (poor sleep vs stress).
Longitudinal or time-series design, and include pre and post-test
Selection
Internal validity
differences in respondent characteristics ( young vs. old)
Random assignment
History
Internal Validity
concurrent events (political events or laws)
Use a control group to compare the effect of external events on both the treatment and control groups.
Maturation
Internal validity
natural changes (older and aging)
Shorten the time span of the study
Regression
Internal validity
extreme scores reverting to mean (high stress ppl, lower level report)
Use repeated measures and include a wide range of participants to reduce regression to the mean effects.
Attrition
Internal validity
systematically drop out (a demanding treatment group)
Implement strategies to maintain participant engagement, monitor reasons for dropouts, and use intention-to-treat analysis
T-test
Testing
Internal validity
test exposure (2nd test with familiarity)
Employ alternative forms of the test or counterbalance the test forms to reduce the impact of repeated testing
Instrumentation
Internal validity
measure/condition changes over time (change in surveys)
Ensure consistency in measurement tools and procedures throughout the study, and calibrate instruments if necessary.
Additive and interactive effects
Internal validity
combined or dependent multiple threats (history + maturation)
Design the study to isolate variables and use statistical controls to account for potential interactions and additive effects.
Interaction of causal relationships with unit
External validity
variability with different unit types (effective in adults NOT in children)
Use random sampling to select participants and replicate the study across different populations.
Interaction of causal relationships over treatment variations
External validity
variability with different treatments (effects of different dosages)
Vary the levels of the treatment and replicate the study with different treatment protocols.
Interaction of causal relationships with outcomes
External validity
variability with different outcome observations (improved mood but not cognitive function from a treatment)
Measure multiple outcomes and use different methods to assess the outcomes to ensure generalizability.
Interaction of causal relationships with settings
External validity
variability in different settings (therapy effective in clinical settings but not at home)
Conduct the study in various settings or replicate the study in different environments to test for consistency.
Context-dependent mediation
External validity
mediator variability in different contexts (social support mediating stress reduction only in community setting)
Explore the mediating effects in different contexts and include diverse contexts in the study design to assess the robustness of the mediating effect.
Inadequate explication of constructs
Construct Validity
poor construct definition lead to incorrect inferences (vaguely defined well-being in psychological.)
Clearly define constructs and ensure they align with established theories; use a panel of experts for validation.
Construct confounding
Construct validity
failure to describe all relevant constructs (not consider both anxiety and depression in mental health study)
Distinguish the constructs from one another and measure them independently; use confirmatory factor analysis.
Mono-operation bias
Construct Validity
limited operationalization complicating inference (only use self-reports to measure happiness)
Employ multiple measures or observations for the construct to capture its complexity.
Mono-method bias
Construct validity
only use one method, becomes part of the studied construct (self-report)
Use different methods of measurement (e.g., observations, interviews, and surveys) to assess the construct.
Confounding construct with levels of constructs
Construct validity
failure to describe the limited levels of studied construct (no specific type of aggression in a study on aggressive behavior)
Define constructs at consistent levels of specificity and avoid mixing abstract and concrete levels.
Treatment-sensitive factorial structure
Construct validity
measured structure changing due to treatment, hidden by consistent scoring.
Perform factorial analysis both before and after the treatment to ensure the treatment does not alter the construct.
Reactive self-report changes
Construct Validity
self-report changes by participant motivation changes (participants provide fakeanswers)
Utilize non-reactive measures and triangulate self-report data with objective data when possible.
Reactive to the experimental situation
Construct validity
hypothesis guessing
ppl response reflecting perceptions of experimental situation (ppl know they are in the experimental and behave differently)
Design experiments to be as naturalistic as possible and use field experiments to reduce this reactivity.
Experimenter expectancies
Construct validity
experimenter influence on ppl responses
Use double-blind procedures where neither the participant nor the experimenter knows the treatment condition.
Novelty and disruption effect
Construct validity
unusual response to novel and disruptive treatment (positive response to a new tech just because it is new)
Familiarize participants with the experimental process to reduce the novelty factor.
Compensatory equalization
construct validity
compensatory goods to non-treatment group (control group), to make sure control group received equalized compensatory
Prevent control groups from knowing what the experimental group receives to avoid compensatory behavior.
Compensatory rivalry
Construct validity
control group motivated to perform as well as those treated (extra efforts from control group to match treatment group)
Keep participants blind to other groups’ conditions to reduce competitive efforts to compensate.
Resentful demoralization
Construct Validity
negative response from ppl due to resentment (control group shows less engagement due to not receiving treatment)
Ensure equal treatment of groups in terms of interactions and communications to prevent demoralization.
Treatment diffusion
Construct validity
treatment received by non-assigned participants (treatment information known by ppl in control group)
Isolate groups and maintain confidentiality about the specific nature of treatments to prevent sharing of treatment details.
Low statistical power
Statistical conclusion validity
Insufficient experiment power leading to incorrect conclusions about significance
Increase the sample size or use more sensitive statistical methods.
Violated assumptions of statistical test
Statistical conclusion validity
Violations leading to overestimation or underestimation of effect size and significance
Use nonparametric tests or transform data to meet assumptions.
Fishing and error rate problem
Statistical conclusion validity
Repeated testing inflates statistical significance if not adjusted.
Apply corrections like the Bonferroni adjustment for multiple comparisons.
Unreliability of measures
Statistical conclusion validity
Measurement error affecting the relationships between variables.
Use validated and reliable instruments; conduct pilot studies if necessary.
Restriction of range
Statistical conclusion validity
Reduced variable range weakening relationships.
Ensure sampling covers the full range of the variable.
Unreliability of treatment implementation
Statistical conclusion validity
Partial or inconsistent treatment implementation affecting effect estimation.
Standardize treatment protocols and training.
Extraneous variance in the experimental setting
Statistical conclusion validity
Features of the setting inflating error and obscuring effects.
Control extraneous variables and use randomization.
Heterogeneity of units
Statistical conclusion validity
Increased variability within conditions raising error variance.
Use matching or statistical controls to account for participant variability.
Inaccurate effect size estimation
Statistical conclusion validity
Statistics overestimating or underestimating effect sizes.
Use appropriate statistical methods and provide confidence intervals.
Hallmark of good study
1 variable 1 time.
Isolate changes! With 2 more variation, it is difficult to know what is the casual factor
Double-blind procedure
What? This is a construct validity. Try to make sure the participant don’t know or cannot guess the hypothesis. Generally used in clinic trials to avoid bias. No one (participants or researchers) should know who receive the actual treatment and who receive placebo.
How to deal with? In the planning and setup to make sure it is indistinguishable to both participants and researchers. Implication such as same color, taste, equal treatment will help.
Proxy variable
What? Using a different variable in place of another variable as a proxy variable – indirect measurement.
Whether it is appropriate? (1) High correlation (2) whether it is used before by building theoretical argument to build a plausible argument.
Moderation vs mediation
(1) Mediation: how variable A and B related through C, A->C->B
(2) Moderation How variable A and B are related with the presence of C, A->B (C)
Two groups and with group variance, Intervention
F-value=(Between-group variance)/(Within-group variance)
Need more between group variance than within group variance. to maximize the F-values by (1) enlarging between-group variance and/or (2) reducing the within-group variance. By using either experimental or statistical controls to archive it.
Ceiling effects or flooring effects
What? Most of the score a located on the higher ranges, e.g., from 0 to 10 ranges, it sits between 7 and 10. Plot the data.
Two threats to the validity
(1) Statistical threat: Range restriction.
(2) Confounding threat: the construct validity:
Solomon 4-group
O1 pre test
O2 post test
X treatment
Random test vs quasi-random test
possible vs plausible threat to validity
possible threats” refers to any and all threats that could theoretically occur, given the circumstances and design of the study. – random dropout
“Plausible threats,” on the other hand, are those that not only could occur but also have a reasonable likelihood of occurring based on the specific context of the study. – systematic dropout
Determining the presence of compensatory rivalry or resentful demoralization as a threat to construct validity based on visual inspection of data.
construct validity
compensatory rivalry: control group behave better on purpose
ressentful demoralization: control group behave negatively.
History vs. Maturation
internal validity
History threat is out of researchers’ control, e.g. new law.
Maturation threat is the change over time for participants. To design a shorter test, if possible.
Content vs. Construct Validity
Content validity is about whether the content can capture the concept fully.
Construct validity is about the method or operation that can capture the measure of the test.
Proxy Measure
A proxy measure is an indirect measure of an outcome when direct measurement is not possible. It’s used when the actual variable of interest cannot be easily or ethically measured, so a related indicator is measured instead, with the assumption that it closely approximates the true variable
Confounding Constructs with Levels of Constructs
construct validity
Confounding construct: the struct has confounding effects and not clearly defined, which causes the errors in the measurement
Level of construct, there are different levels of construct that are not design properly causing measuring errors.
Exposure in pretest
To establish whether this is a plausible threat, researchers can use a Solomon Four-Group design. This design includes four groups: two of them receive the pretest, and two do not. Of the two that receive the pretest, one group receives the treatment and the other does not. Of the two that do not receive the pretest, one group receives the treatment and the other does not. By comparing the groups, researchers can determine if the pretest influenced the outcomes.
