Module 4-Slides

Question 1

Design Validity
(applicable to quantitative study)

Answer 1

Relates to the truthfulness or accuracy of study results (Believability)

External Validity
Construct Validity
Internal Validity (for intervention studies/explanatory research)
Statistical Conclusion Validity

*Each form is cumulatively dependent on the components below it

Question 2

Explanatory Research GOALS

Answer 2

Corresponds to 4 types of design validity

Question 3

The PURPOSE of design Validity and type of Design is…

Answer 3

to examine issues of threats that must be controlled in the design and analysis of research

Question 4

Statistical Conclusion Validity

Answer 4

Appropriate use of statistical procedures to assess the relationship between the IV and DV
Potential threats:
-Low statistical power (small sample size)
-Violated assumptions of statistical tests (level of measurement, frequency distribution)
-Reliability and variance (unreliable measure)
-Failure to use “intention to treat (ITT) analysis”

Question 5

Intention to treat (ITT) (not Tx but statistical analysis)

Answer 5

Usually described a “once randomized, always analyzed”
Includes every subject who is randomized according to randomized Tx assignment
Ignores noncompliance, protocol deviations, withdrawal, and anything that happens after randomization
Maintains baseline/prognostic balance generated from the original random allocation or assignment

Question 6

Internal Validity

Answer 6

The extent to which experimental Tx (IV) really caused the observed change in the outcome measures (DV)

Potential for confounding factors (alternative causes) to interfere with the relationship between IV andDV

Question 7

Internal Validity - Internal Threats

AHA TIMS

Answer 7

History
Maturation
Attrition
Testing
Instrumentation
Regression to the mean
Selection (assignment)

Question 8

Internal Validity - Social Threats

Answer 8

Diffusion or imitation
Compensatory equalization
Compensatory rivalry
Demoralization

Question 9

History - Did unanticipated events occur during the study that could affect the DV?

Answer 9

Nature of Threats:
events unrelated to the Tx of interest that influence the study’s outcome
*confounding effect of specific events, other than the experimental intervention, that occur after the introduction of the IV or between a pretest and posttest

Solution:
Control group and random assignment: to separate the effect of history
Time or Plan the study to avoid a predictable event
Apply statistical adjustment (ANCOVA)

Question 10

Maturation - Were changes in the DV due to normal development or the simple passage of time?

Answer 10

Threat:
changes in human behavior or function OVER THE COURSE OF TIME that may influence the study results

Solution:
Use random assignment
Randomize the testing or Tx order
Apply statistical adjust.

Question 11

Attrition (aka Drop-out/Mortality) - Is there a differential loss of subjects across groups?

Answer 11

Threat:
loss of subjects causes a reduction in sample size
cause imbalanced baseline characteristics

Solution:
Replacement of lost subjects if possible
Re-examination of data to see existing status
Statistical ITT analysis

*NO RANDOM ASSIGNMENT

Question 12

Testing - Did the pretest or repeated test affect scores on the posttest?

Answer 12

Threat:
repeated testing on subjects that may lead to increased familiarly or inappropriate cues provided by the tester (construct validity)

Solution:
Elimination of multiple testing
Use of randomly selected experimental and control groups

Question 13

Instrumentation - Did the DV change because of how it was measured?

Answer 13

Threat:
Problems w/the tool used to measure the variables of interest
Inappropriate selection of the tool (Measurement)

Solution:
Select appropriate technique or instrument
Calibrate instrument
User training or practice in taking measurement

Question 14

Regression to the Mean - Is there evidence of regression from pretest to posttest scores?

Answer 14

Threat:
occurs when subjects enter a study with an extreme value (outlier) for BASELINE of outcome measure

Solution:
trim outliers
take repeated baseline measures
use a CONTROL GROUP

Question 15

Assignment (Selection) - Is there bias in the way subjects have been assigned to experimental groups?

Answer 15

Threat:
Unequal baseline characteristics of groups that might influence the study’s outcome

Solution:
Random assignment to groups
Define inclusion and exclusion criteria adequately
Statistical adjustments (ANCOVA)

Question 16

Construct Validity

Answer 16

How variables are conceptualized
Concerned w/ the MEANING OF VARIABLES w/in a study, and whether the IV/DV are well-established and correctly labeled
Often related to measure/test: to compare the VARIABLES with THEIR MEASURES to determine if the measure truly represents the variable
->Ex: shld function vs active ROM

Question 17

Construct Validity Threats

Answer 17

regarding OD
Time frame: length of study
Multiple Tx interactions
Experimental bias related to subjects/investigators (Blinding/Masking)

Question 18

Hawthorne Effect

Answer 18

One possible type of experimental bias
The effect of subject’s knowledge that they are part of a study on their performance

Question 19

External Validity

Answer 19

Extent to which the results of a study may be generalized to outside experimental situation (other individuals or circumstances)

Question 20

External Validity Threats

Answer 20

1. Biased sample selection (influence of selection)
   -only a narrowly defined subset of subjects/persons
   -only woman not man or vice versa
2. Setting differences (influence of setting)
   -env. of study was different from what would be applied in clinic
3. Time (influence of history)
   -considerably changed circumstances (updated practice guideline)