Chapter 15 validity of design Flashcards
statistical conclusion validity
Is there an actual relationship between the I and D variables?
statistical conclusion validity: statistical power
its ability to document a real relationship between I and D variables?
A study with low Power may not be able to establish this
-usually due to low sample size may not provide a statistical difference
-while a very large sample size may contribute to showing a statistical difference
statistical conclusion validity: violated assumptions of statistical tests
the assumptions that exist, if not met, may lead to erroneous inferences
statistical conclusion validity: reliability and Variance
statistical conclusions are messed up by certain things as unreliable measurement, failure to standardize the protocol, environmental interferences, or heterogeneity of subjects
statistical conclusion validity: Failure to use Intention to Treat Analysis
when subjects go missing or do not finish the research, the ITT helps to control for this
Internal Validity :
given a statistical relationship between the I and the D, is there evidence that one causes the other?
Internal Validity : T/F The assumption of causality or cause and effect require three components: temporal precedence, covariation of cause and effect, and no plausible alternative explanations.
True
Internal Validity: temporal precedence
the cause, precedes the effect
internal validity: covariation of cause and effect
the outcome only occurs in the presense of the intervention
Internal validity: no plausible alternative explanations
alternative explanations not plausible
Internal vs social threats
internal threats of validity of design are events of changes that occur during a study that may affect the relationship between the I and the D
Social threats of validity of design are the pressures that can occur in research that may lead to differences between groups
i.e. (performance bias where one group is aware of the other group)
Internal threats
history
Maturation
Attrition
Testing
Instrumentation
Regression to the Mean
Internal Threats : History
Did unanticipated events occur during the study that could affect the dependent variable?
Safeguard for History
Random assignment
Time or plan the study better to avoid predictable event
Apply statistical adjustment
- If participants were also involved in other activities or received
other type of intervention (e.g., OT), this is an example of what type of internal threat?
it is an example of History
threat that interferes with the study Results
Internal threat :Maturation
Were changes in the dependent variable due to normal development or the simple passage of time?
Child growing up, person getting older, spontaneous recovery post stroke
Possible solution / safeguard for Maturation?
- Use random assignment
- Apply statistical adjustment
- Randomize the testing or treatment order
Internal threat : attrition / experimental mortality
Is there a differential loss of subjects across groups?
Studies with long follow ups are prone to Attrition.
Safeguard for Attrition?
- Replacement of lost subjects if possible
- Re-examination of data to see existing status;
- Statistical “intention-to-treat” (ITT) analysis
Internal threat : testing
Did the pretest or repeated test affect scores on the posttest?
Repeated testing on subjects may lead to increased familiarity of inappropriate cues provided by the tester
Solution or safeguard for TESTING?
o Elimination of multiple testing;
o Use of randomly selected experimental and
control groups;
Internal threat : Instrumentation
Did the dependent variable change because of how it was measured?
Problems with the tool used to measure the variables of interest
Inappropriate selection of the tool
Safeguard/solution to instrumentation?
o Select appropriate technique or instrument
o Calibrate instrument
o User training or practice in taking measurement
Internal threat : Regression to the Mean
Is there evidence of regression from pretest to posttest
scores?
Occurs when subjects enter a study with an extreme value (outlier) for baseline of outcome measure
Solution/safeguard for regression to the mean?
o To trim outliers;
o To take repeated baseline measures;
o To use a control group
Internal threat: Assignment / selection
Is there bias in the way subjects have been assigned to experimental groups?
Unequal baseline characteristics of groups that might influence the study’s outcome.
Possible solution/safeguard for assignment
o Random assignment to groups;
o Adequately defined inclusion and exclusion
criteria;
o Statistical adjustments e.g., analysis of
covariance (ANCOVA)
AHA TIMS
- Attrition
- History
- Assignment
- Testing
- Instrumentation
- Maturation
- Statistical regression to mean
Ruling Out Threats to internal threats of the Internal
Validity
▪Random assignment (and control groups) may
control threats due to
✓history,
✓maturation,
✓Assignment/selection,
✓regression to the mean,
✓testing,
✓instrumentation
Construct Validity
▪Construct validity, in the context of design, concerns how variables are conceptualized.
▪is concerned with the meaning of variables within a study, and whether the IV/DV are well-established and correctly labeled, specifically.
▪ often (but not exclusively) related to measure/test:
to compare the variables with their measures to
determine if the measure truly represents the variable, e.g.,
➢ shoulder function vs. active ROM
Construct Validity
▪ Threats can occur regarding
* Operational definitions (OD)
* Time frame: length of a study
* Multiple treatment interactions: Tx, other than the one of interest, is
administered, e.g., Parkinson medication vs. deep brain stimulator (DBS)
* Experimental bias related to subjects/investigators (Refer to previous
module on blinding/masking as strategy)
✓ Subjects’ performance bias;
✓ Investigators’ bias toward a particular outcome
“Hawthorne Effect”
- One possible type of experimental bias
- The effect of subjects’ knowledge that they are part of a
study on their performance
➢ First described related to a series of experiments on workers’ performance
➢ However a number of flaws in the original experiments
External Validity
▪is the extent to which the results of a study may be generalized to outside experimental situation, e.g., other individuals or
circumstances
Threats to External Validity
- Biased sample selection (influence of selection) - only a narrowly defined subset of subjects/persons; - only woman not man or vice versa
- Setting differences (influence of setting)
- environment of the study was different from what would be applied in clinic - Time (influence of history)
– considerably changed circumstances (e.g., updated practice guideline)
Interesting point from book
- As consumers of research evidence, we are responsible for making
judgments about the degree of internal validity of a study, and the
extent to which we consider findings relevant and useful. - When internal validity is severely threatened, conclusions can be
considered suspect. We might assume that peer review would
prevent publication of such studies, but that is not necessarily the
case—caveat emptor!
“Intention to treat (ITT)” (not Tx but statistical analysis)
- ITT analysis is usually described as “once randomized, always analyzed”.
- ITT analysis includes every subject who is randomized according to randomizedtreatment assignment.
- It ignores noncompliance, protocol deviations,
withdrawal, and anything that happens after randomization. - ITT analysis maintains baseline/prognostic balance
generated from the original random allocation or
assignment (see slide #9).
Statistical Conclusion Validity
▪Appropriate use of statistical procedures to assess the
relationship between the independent and dependent variables (IV and DV)
▪ Potential threats
* Low statistical power (e.g., small sample size)
* Violated assumptions of statistical tests
✓ level of measurement, frequency distribution
* Reliability and variance (unreliable measure)
* Failure to use “intention to treat (ITT) analysis”
Types of design validity
Statistical conclusion validity (relationship)
Internal validity (evidence for cause and effect / experimental )
Construct validity (what parts can be generalized)
External validity (can the results be further generalized, persons setting, or times)
Step 3: Appraise the Literature
Three primary questions:
▪ Is the study valid?
▪ Are the results meaningful?
▪ Are the results relevant to my patient?
