W10: Systematic Reviews and Meta-Analysis Flashcards
What is a systematic review?
A research project that seeks to give an overview of the totality of research into a specific topic
Why do we need systematic reviews? (4)
- Vast literatures
- Primary studies are often contradictory
- Hard to identify small but important effects
- Narrative reviews ( Not replicable, prone to bias, can still be contradictory)
Quality of research pyramid
Systematic review: Searching the literature (2)
- Attempting to get all avaliable evidence (including “grey literature and foreign language studies)
- Will require multiple electronic databases, hand searches of key journals, reference and cited by lists, contacting stakeholders, research instititues and authors
What is grey liteature? - (2)
- Typical academic literature is published in peer-reviewed literature
- There is a lot of stuff that does not get published in peer-reviewed literature like researcher discussing preliminary findings in a study presented in a conference (grey literature)
Systematic review: we search literature and define search stratgery based on
PICOS framework
PICOS framework may not be relevant to
observational studies
PICOS framework - Population
– e.g., offenders, students, patients with depression
PICOS framework - Intervention
CBT, rapport
PICOS framework - Comparator
usual care, interrogation
PICOS framework - Outcomes
e.g., hallucinations, information provided
PICOS framework - Study design
RCTs, longitudinal studies, field studies
Systematic review - Search literature using Boolean logic
AND
Captures all the relevant PICOS elements
Systematic review - Search literature using Boolean logic
OR
To remain specific relevant evidence with each element
Systematic review - Search literature using Boolean logic
We check stratgeries of similar reviews to
improve on search stratgery
Systematic review - Search literature using Boolean logic
Don’t neglect - (2)
regional differences and time-period differences
e.g., Dyspraxia (UK), Developmental Coordination (USA)
Systematic review: How to find grey literature? (3)
- Ask prominent authors
- Conference proceedings
- Research group websites/charities
When searching literature with systematic review we should document
why should we document (2)
everything (e.g., how many articles were identified, how many were duplicates)
This is for transparency when things go wrong and need to be able to reproduc precisely
Systematic review study selection: Based on inclusion and exclusion criteria all articles would be screened by
title and abstract by two individuals and then by full text.
Systematic review: Study selection - the inclusion criteria should possibly link with the
PICOS criteria
Systematic review: Study selction: Still document everything (3)
- Number of articles identified and excluded at each stage
- Reasons for exclusion
- Include a flowchart of studies
All relevant information would be recorded (After study selection, inclusion, exclusion)
recorded on data extraction forum
Systematic review - extracting the data
Full systematic review require two reviewers to (3)
- Permit rapid data extraction
- Facilitate fast comprehension of key points
- Contain enough information for synthesis
When recording the data in systematic review we want (4)
generic study info: author, year, journal
Sample info: nationality, occupations, age
Methods: study design, procedure, setting, length of follow up
Outcomes: effect size, measures of spread, sample sizes
When recording data in systematic review what do we do for missing data? (2)
Contact authors
Sensitivity analyses
Individual studies have flaws, bias leads to effects being
over or underestimated
Sometimes there is an obligation (in systematic review) to
assess the risk of bias in studies
What an issue of measuring study quality scales in systematic review? (2)
Study quality scales are generally poor
Summary scores are not particularly informative
Many systematic reviews do not include
statistical combiniation
Systematic reviews when it comes to data synthesis tabulate details from all studies to assess a body of evidence by doing (2)
thorough analysis of differences between studies
e.g., effects of different samples, measures, settings
What is meta-analysis?
Mathematical combination of study results to make a combined estimate of population effect
Some articles may have effect size in Pearson’s r and another in Cohen’s d
THne we need to
Convert to the same metric
Meta-Analysis weighted average of all study estimates
e.g,, Group 1 had average 34 test score and 20 people in it
Group 2 had average 36 test score and had 30 people in it
Calculate weighted average
2 different ways we do meta -analysis (weighted by precision ; SE)
Fixed effects
Random effects
Fixed effect formula
w = 1 / (se)^2
Random effects formula
w = 1/ (se)^2 + t
Fixed effects MA (meta-analysis) assumes
all variability between studies is due to sampling error
Random effects (MA) assumes that there is
sampling error + genuine differences between study populations
We would use random effects when there is for example (2)
Often times greater hetrogenity than sampling error is some studies use questionnaire 1 while others use different questionnaries
1st questionnaire find bigger effect size than seond questionnaire
In psychology, we usually use random effects unless (2)
- You are analysing direct replications
- You have a very small numberof studies with very different sample sizes
We have another metric named ‘t’ (tau) in random effects analysis as it is
extra variance beyond that accounted for by study imprecision, i.e.,
between study error
New development of instead needing systematic reviews and meta-analysis
Massive multi-lab studies running the exact same procedures,
E.g., Psychological Science Accelerator
Advantages of multi-lab/multi-site replications over meta-analysis - (2)
Low heterogeneity
No publication bias
Disadvantages of multi-lab/multi-site replications over meta-analysis - (2)
Sometimes the heterogeneity is the important bit
Even narrower in scope, so limited to the most important (and simplest?) research questions
Heterogenity analysis aim in random effects is to
find sources of systematic variation
Aim of heterogenity is to specifically identify two things (2)
- Identify reliable and unreliable measures
- Identify differences due to samples
What is heterogenity analysis done via? (2)
- Subgroup analyses
- Meta-regression
Issues of heterogenity analysis - subgroup and meta-regression - (2)
- Tend to have lower power (both require at least 10 stuides)
- Multiple comparisons - specificy and justify comparisons in advance when possible
Subgroup analyses steps - (2)
Divide studies according to a particular feature (e.g., students vs general population, study designs like prospective vs retrospective and measures of DVs)
Run analyses separately - any differences in outcome?
Meta-regression (2) steps
Linear regression weighted by study precision
Assess impact of explanatory variables upon estimate of effect size (E.g., mean age, length of follow-up)
Asymmtery in funnel plot (2)
- There is a gap of missing studies in plot
- Often what happens when publication bias happens as studies who don’t get effect don’t get published
Suppose that we did not have a suitable estimate of the standard
deviation available. Explain how a systematic review could be used
to obtain one (5)
In a systematic review we would consult multiple electronic databases,
searches of key journals, reference and cited by lists, and contact stakeholders to identify works reporting standard deviations
of 400m times.
The PICOS framework would be used and searches made via suitable Boolean logic, e.g., “standard” and “deviation” and “400m”.
Based on inclusion and exclusion criteria all articles would be screened
by title and abstract by two individuals and then by full text.
All relevant information would be recorded on data extraction
forms.
The critical details of the studies, including the standard
deviation estimates would then be tabulated to provide the body
of evidence for selection of a suitable value.
