Chapter 15 validity of design

Question 1

statistical conclusion validity

Answer 1

Is there an actual relationship between the I and D variables?

Question 2

statistical conclusion validity: statistical power

Answer 2

its ability to document a real relationship between I and D variables?
A study with low Power may not be able to establish this
-usually due to low sample size may not provide a statistical difference
-while a very large sample size may contribute to showing a statistical difference

Question 3

statistical conclusion validity: violated assumptions of statistical tests

Answer 3

the assumptions that exist, if not met, may lead to erroneous inferences

Question 4

statistical conclusion validity: reliability and Variance

Answer 4

statistical conclusions are messed up by certain things as unreliable measurement, failure to standardize the protocol, environmental interferences, or heterogeneity of subjects

Question 5

statistical conclusion validity: Failure to use Intention to Treat Analysis

Answer 5

when subjects go missing or do not finish the research, the ITT helps to control for this

Question 6

Internal Validity :

Answer 6

given a statistical relationship between the I and the D, is there evidence that one causes the other?

Question 7

Internal Validity : T/F The assumption of causality or cause and effect require three components: temporal precedence, covariation of cause and effect, and no plausible alternative explanations.

Answer 7

True

Question 8

Internal Validity: temporal precedence

Answer 8

the cause, precedes the effect

Question 9

internal validity: covariation of cause and effect

Answer 9

the outcome only occurs in the presense of the intervention

Question 10

Internal validity: no plausible alternative explanations

Answer 10

alternative explanations not plausible

Question 11

Internal vs social threats

Answer 11

internal threats of validity of design are events of changes that occur during a study that may affect the relationship between the I and the D
Social threats of validity of design are the pressures that can occur in research that may lead to differences between groups
i.e. (performance bias where one group is aware of the other group)

Question 12

Internal threats

Answer 12

history
Maturation
Attrition
Testing
Instrumentation
Regression to the Mean

Question 13

Internal Threats : History

Answer 13

Did unanticipated events occur during the study that could affect the dependent variable?

Question 14

Safeguard for History

Answer 14

Random assignment
Time or plan the study better to avoid predictable event
Apply statistical adjustment

Question 15

• If participants were also involved in other activities or received
  other type of intervention (e.g., OT), this is an example of what type of internal threat?

Answer 15

it is an example of History
threat that interferes with the study Results

Question 16

Internal threat :Maturation

Answer 16

Were changes in the dependent variable due to normal development or the simple passage of time?
Child growing up, person getting older, spontaneous recovery post stroke

Question 17

Possible solution / safeguard for Maturation?

Answer 17

• Use random assignment
• Apply statistical adjustment
• Randomize the testing or treatment order

Question 18

Internal threat : attrition / experimental mortality

Answer 18

Is there a differential loss of subjects across groups?
Studies with long follow ups are prone to Attrition.

Question 19

Safeguard for Attrition?

Answer 19

• Replacement of lost subjects if possible
• Re-examination of data to see existing status;
• Statistical “intention-to-treat” (ITT) analysis

Question 20

Internal threat : testing

Answer 20

Did the pretest or repeated test affect scores on the posttest?
Repeated testing on subjects may lead to increased familiarity of inappropriate cues provided by the tester

Question 21

Solution or safeguard for TESTING?

Answer 21

o Elimination of multiple testing;
o Use of randomly selected experimental and
control groups;

Question 22

Internal threat : Instrumentation

Answer 22

Did the dependent variable change because of how it was measured?
Problems with the tool used to measure the variables of interest
Inappropriate selection of the tool

Question 23

Safeguard/solution to instrumentation?

Answer 23

o Select appropriate technique or instrument
o Calibrate instrument
o User training or practice in taking measurement

Question 24

Internal threat : Regression to the Mean

Answer 24

Is there evidence of regression from pretest to posttest
scores?
Occurs when subjects enter a study with an extreme value (outlier) for baseline of outcome measure

Question 25

Solution/safeguard for regression to the mean?

Answer 25

o To trim outliers;
o To take repeated baseline measures;
o To use a control group

Question 26

Internal threat: Assignment / selection

Answer 26

Is there bias in the way subjects have been assigned to experimental groups?
Unequal baseline characteristics of groups that might influence the study’s outcome.

Question 27

Possible solution/safeguard for assignment

Answer 27

o Random assignment to groups;
o Adequately defined inclusion and exclusion
criteria;
o Statistical adjustments e.g., analysis of
covariance (ANCOVA)

Question 28

AHA TIMS

Answer 28

• Attrition
• History
• Assignment
• Testing
• Instrumentation
• Maturation
• Statistical regression to mean

Question 29

Ruling Out Threats to internal threats of the Internal
Validity

Answer 29

▪Random assignment (and control groups) may
control threats due to
✓history,
✓maturation,
✓Assignment/selection,
✓regression to the mean,
✓testing,
✓instrumentation

Question 30

Construct Validity

Answer 30

▪Construct validity, in the context of design, concerns how variables are conceptualized.
▪is concerned with the meaning of variables within a study, and whether the IV/DV are well-established and correctly labeled, specifically.
▪ often (but not exclusively) related to measure/test:
to compare the variables with their measures to
determine if the measure truly represents the variable, e.g.,
➢ shoulder function vs. active ROM

Question 31

Construct Validity

Answer 31

▪ Threats can occur regarding
* Operational definitions (OD)
* Time frame: length of a study
* Multiple treatment interactions: Tx, other than the one of interest, is
administered, e.g., Parkinson medication vs. deep brain stimulator (DBS)
* Experimental bias related to subjects/investigators (Refer to previous
module on blinding/masking as strategy)
✓ Subjects’ performance bias;
✓ Investigators’ bias toward a particular outcome

Question 32

“Hawthorne Effect”

Answer 32

• One possible type of experimental bias
• The effect of subjects’ knowledge that they are part of a
  study on their performance
  ➢ First described related to a series of experiments on workers’ performance
  ➢ However a number of flaws in the original experiments

Question 33

External Validity

Answer 33

▪is the extent to which the results of a study may be generalized to outside experimental situation, e.g., other individuals or
circumstances

Question 34

Threats to External Validity

Answer 34

1. Biased sample selection (influence of selection) - only a narrowly defined subset of subjects/persons; - only woman not man or vice versa
2. Setting differences (influence of setting)
   - environment of the study was different from what would be applied in clinic
3. Time (influence of history)
   – considerably changed circumstances (e.g., updated practice guideline)

Question 35

Interesting point from book

Answer 35

• As consumers of research evidence, we are responsible for making
  judgments about the degree of internal validity of a study, and the
  extent to which we consider findings relevant and useful.
• When internal validity is severely threatened, conclusions can be
  considered suspect. We might assume that peer review would
  prevent publication of such studies, but that is not necessarily the
  case—caveat emptor!

Question 36

“Intention to treat (ITT)” (not Tx but statistical analysis)

Answer 36

• ITT analysis is usually described as “once randomized, always analyzed”.
• ITT analysis includes every subject who is randomized according to randomizedtreatment assignment.
• It ignores noncompliance, protocol deviations,
  withdrawal, and anything that happens after randomization.
• ITT analysis maintains baseline/prognostic balance
  generated from the original random allocation or
  assignment (see slide #9).

Question 37

Statistical Conclusion Validity

Answer 37

▪Appropriate use of statistical procedures to assess the
relationship between the independent and dependent variables (IV and DV)
▪ Potential threats
* Low statistical power (e.g., small sample size)
* Violated assumptions of statistical tests
✓ level of measurement, frequency distribution
* Reliability and variance (unreliable measure)
* Failure to use “intention to treat (ITT) analysis”

Question 38

Types of design validity

Answer 38

Statistical conclusion validity (relationship)
Internal validity (evidence for cause and effect / experimental )
Construct validity (what parts can be generalized)
External validity (can the results be further generalized, persons setting, or times)

Question 39

Step 3: Appraise the Literature

Answer 39

Three primary questions:
▪ Is the study valid?
▪ Are the results meaningful?
▪ Are the results relevant to my patient?