Evaluation Designs Flashcards
What are the 3 main stages of evaluation?
Formative
Process
Outcome
Describe the formative stage of an evaluation
Happens before any intervention to evaluation.
Tests the acceptability + feasibility of the intervention.
Mainly qualitative i.e focus groups and in-depth interviews
Describe the process stage of an evaluation
Happens whilst the intervention is underway.
Measures how the intervention was derived + received.
Mixed quantitative and qualitative
Describe the outcome stage of an evaluation
Measures whether the intervention has achieved its objectives.
Mainly quantitive
What is the main purpose of an evaluation design?
To be as confident as possible that any observed changes were caused by the intervention, rather than by chance or other unknown factors.
List the criteria for inferring causality
Cause must precede the effect
Plausibility
Strength of the association
Dose-response relationship
Reversibility
Criteria for inferring causality
How is the strength of the association measured
Effect size
or
Rel. risk
Criteria for inferring causality
What comes under the dose-response relationship
Occurs when changes in the level of a possible cause are associated with changes in the prevalence or incidence of the effect.
Criteria for inferring causality
What is meant by reversibility?
When the removal of the possible cause results in a return to baseline for the outcome.
What does high internal validity mean
High means the differences observed between the groups are related to the intervention tested in the trial.
Define external validity
Extent to which the results of an experiment of an intervention can be generalised to the target or general population.
What are the types of Evaluation design
Experimental - Randomly assigned controls or comparison groups
Quasi-Experimental - Not randomly assigned controls or comparison groups
Non-experimental - No comparison or control group
Strengths to experimental evaluation design
Can infer causality with highest degree of confidence
Weaknesses to experimental evaluation design
Most resource intensive of the evaluation designs
Req ensuring minimal extraneous factors
Can sometimes be challenging to generalise to the “real world”
Strengths to the quasi-experimental evaluation design
Can be used when unable to randomise a control group but still allows comparison across groups +/or time
Weaknesses to the quasi-experimental evaluation design
Differences between comparison groups may be confound
Group selection is critical
Moderate confidence in inferring causality
Strengths to the non-experimental evaluation design
Simple
Used when baseline data +/or comparisons groups are not available
Good for a descriptive study
May req fewer resources
Weakness to the non-experimental evaluation design
Minimal ability to infer causality
What are the types of RCT (Experimental design)
Randomised cross-over trials
Parallel randomised trials
What is the purpose of random assignment
To best ensure the intervention is the only difference between the 2 groups.
To ensure any factors influencing the outcome are evenly distributed between the groups.
List the main threats to internal validity in RCT (Experimental designs)
Selection bias
Performance bias
Detection bias
Attrition bias
Random Error
Define selection bias
When individuals in both groups differ systematically on a factor that may affect the outcome thereby leading to a systematic error in outcome.
What can decrease selection bias
Randomisation and a matched control group.
Define performance bias
Occurs if there’s insufficient adherence to the study protocol by researchers or participants.
i.e researchers may not deliver the intervention consistently to all participants and participants may differ in how they adhere to the intervention.
Define detection bias
Researchers can administer outcome measures differently between groups.
Participant receiving an intervention they like may over report changes in behaviour.
How can performance and detection bias be avoided?
By blinding researchers
+/or
Blinding participants to group allocation
Define attrition bias
Systematic differences in the number of drop outs from the study between intervention and control groups.
What can attrition bias lead to?
Systematic differences in groups at follow up if its not balanced.
How can attrition bias be managed?
By intention to treat analysis methods
Advantages to RCT (Experimental design)
Can be most confident that any observed changes can be attributed to the intervention and not any other factors.
Allows randomisation of participants to both groups + concealment of their allocation ensures selection bias and confounding or unknown variables are minimised
Which experimental design is regarded as having high internal validity
RCT
Disadvantages to RCT (Experimental design)
Expensive
Time consuming
Can have high drop out rates if the intervention has undesirable side-effects or little incentive to stay in the control arm
Ethical consideration may mean research Q can’t be investigated using RCT
Prior knowledge is requires for sample size calculation
Can have issues with generalisability (participants volunteering to participate may not be representative of the population being studies) - Low external validity
Cluster RCT (Experimental Evaluation design option)
When the unit of randomisation is not individuals.
Instead - clusters of individuals in naturally occurring groups.
Is there randomisation in cluster RCT
Yes
Clusters are randomly allocated to the intervention or control group
When are cluster RCT mainly used
When the target of the intervention is the cluster
OR
When its not feasible to prevent contamination in ind RCTs
Advantages to Cluster RCT
Evaluates the real-world effectiveness of an intervention as opposed to efficacy.
Provides an alternative methodology for assessing the effectiveness of interventions in settings where randomisation at the individual level is inappropriate or impossible.
Disadvantages to cluster RCT
Complex + expensive
Req a larger number of ppl vs ind RCT designs due to ppl within clusters potentially being more similar to each other than would be expected by chance.
Getting balanced groups is more difficult - this can decrease internal validity
Analysis is complex
When can quasi-experimental designs be used
When random assignment is NOT possible
Quasi-experimental designs
Controlled before and after intervention design
Same layout as RCT but NO random assignment to groups
Quasi-experimental designs
Controlled before and after intervention design
What could it be at risk from?
Selection bias
What statistical methods are used for quasi-experimental designs?
Difference-in-difference analysis
Regression analysis
What is difference-in-difference analysis used for?
To compare changes before and after the program for individuals in the program + control groups
What is regression analysis used for?
To address the issue of confounding variables by controlling for differences at baseline.
Advantages to Quasi-Experimental Designs
Provides some assurance that outcomes are actually the results of the program
Most practical option for conducting outcome evaluation in community interventions.
Using pre-existing or self-selected groups avoids the additional steps involved w/ randomisation.
Overcomes potential ethical concerns involved in withholding/delaying treatment.
Good for when resources for evaluation are limited
Disadvantages to quasi-experimental designs
Could demand more time
Req access to at least 2 similar groups
W/out randomisation - study groups may differ in important ways that account for some of the group differences in the outcomes after the intervention.
Selection bias
Misclassification or outcome + confounding
What type of experiment does the Interrupted time series design come under?
Quasi-Experimental
Which experimental design is best for overcoming the problems of secular trends?
Interrupted time series design
Advantages of an interrupted time series design
Can detect whether program effects are ST or LT
Series of tests b4 intervention can eliminate need for control group + can be used to project expected results
Can be used if only have 1 study site to conduct evaluation
Can detect secular trends
Disadvantages of an interrupted time series design
Problem of confounding
Changes in instruments during the series of measurements
Loss or change of cases can cause changes in group composition
How can non-experimental designs be strengthened?
By constructing a plausibility argument + controlling for confounding variables
When do you tend to use the Before + after (pre-post) non-experimental design
When you don’t have a comparison or control groups.
Advantages to before + after (pre-post) non-experimental design
Simple
Control for participants prior to knowledge/skills
Disadvantages to before + after (pre-post) non-experimental design
Can’t account for non-program influences on outcomes
Causal attribution not possible
Can’t detect small but important changes
Can’t rule out secular trends
Subject to selection bias
What may happen to the control group in a quasi-experimental design?
May receive different intervention all together
May receive selected components of the intervention being tested
May use a wait-list control: Those in the control don’t receive anything in the study period but will eventually or instead a paired down version once final analysis of trial has been undertaken.
