L7 - Meta Analysis Flashcards
What are the 4 types of review types?
Literature Review
Scoping Review
Systematic Review
Meta-analysis
What are the features of a literature review?
Can be “narrative” or “qualitative”
Not expected to be systematic or overly descriptive in methods
Often Published by ‘experts’ in the field
What are some of the flaws in literature reviews?
Researchers may put emphasis on studies that other researchers see as ‘less important’. Meaning different researchers can reach different conclusions based on the literature.
What are the key features of a scoping review?
Covers key concepts
Sources and types of evidence are available
Explores complex areas or areas not previously reviewed comprehensively or rigorously
What review would you use for the following research questions?
- In what way has the concept of resilience been used in the literature concerning the transition to parenthood?
- To what extent are researchers using theory to underpin their work?
Scoping Review
What are the key features of a systematic review?
- Review that has been conducted in a systematic manner
- Is set-out like other studies: intro, methods, results, discussion, qualatative study of the results of empirical / original research
- Differs from a narrative review: it is comprehensive/exhaustive, rates methodological quality of studies, can include effect sizes.
* Important in evidence based practice, very rigorous and documented methodology. Aim is for the research to be reproduced and updated by other researchers.*
* e.g. looking at the effectiveness of a intervention.*
What review would we use for the following research questions?
How has motivation interviewing been used for adults with multiple sclerosis?
What is the potential impact of delivery format an/or modality on intervention effects?
Systematic Review
What are the key features of a meta analysis
- Statistical analysis of data obtained from empirical studies
- Quantitative data from multiplke studies are combined and analysed
- Main statistic is effect size
- There are different effect sizes - it all depends on the study design/data you need to extract
What is the main goal of a meta-analysis?
To discover how big an effect size actually is,
and what factors might moderate and influence that effect.
What type of review would we use for the following research questions.
How do mean burnout levels reported by postgraduate trainees compare to norms?
Are there differences in burnout patterns, based on training level (interns, residents, registrars), and speciality (e.g. surgical vs. non-surgical)
Meta-analysis
What are the 10 steps of a meta-analysis?
- State Research Question
- Do a literature search
- Decide on objective criteria
- Screen studies
- Extract information
- Assess study quality
- Calculate the effect sizes
- Do the meta-analysis
- Write it up
- Publish
What is the reporting standards set by the APA for meta-analysis called?
MARS
“Meta-analysis Reporting Standards”
The first step to undertaking a meta analysis is to state the research question
What is the criteria we use for stating the research question?
PICO
Population
Intervention (or exposure)
Comparator
Outcome
e.g.
P: adults, both sexes, all ethnicity
I: Mindfulness intervention
C: Psychotherapy
O: Anxiety symptoms or generalized anxiety
Example question: For adults, does mindfulness meditation compared to other psychotherapies reduce anxiety?
What is the most important thing to do when using the PICO criteria when stating the research question
Each step of PICO needs to be defined
If possible, justify the definitions
Step 2 in a meta-analysis is “do a literature search”
How should you go about doing this?
Being a thorough as you can be.
- Search electronic databases that have addressed the same research questions*
- Search conference proceedings, dissertations, Google Scholar etc.*
- Consult experts in the field*
- Search the reference lists of articles you have found*
- Search the reference lists of relevent reviews*
When you are doing your literature review, you will find from 1000 - 10,000 studies.
How do you reduce this into something manageable?
Decide on an objective criteria (step 3 in generating a meta-analysis)
Decide on an inclusion criteria for studies, and you need to justify these criteria, be objective and be explicit
Step 3 in doing a meta analysis is “decide on objective criteria”
What is important to do in deciding on this criteria?
The criteria needs to be justified, objective and explicit.
- example criteria:*
- how have you defined key variables?*
- Which participants are eligible (so reader can assess to whom the results apply)*
- Are specific study designs eligible?*
- Is ther ea time period in which studies needed to be conducted?*
How do you make the process of finding studies easier in a meta-analysis?
Screen Studies (Step 4)
Step 4 of conducting a meta-analysis is to “screen studies”
What steps can you use to do this?
Consider screening tools
How were the studies screened? (i.e. title, abstact, full text)
Number of reviewers involved in search process (if > 1 reviewer, how were disagreements resolved)
Number of studies included
Number of studies excluded (and reasons why)
Step 5 of conducting a meta-analysis is “extract information”
What are some examples of the kinds of information should you be extracting in this phase?
Author name
Year of publication
The population
The study design (RCT? Observational? etc)
The intervention and comparison conditions (if relevant)
The outcomes and how were they measured
No. of individuals (intervention vs. control? Total N?)
Effect size data (e.g. means, SDs, N, r….depends on study design)
Step 6 when conducting a meta-analysis is “assess study quality”
What does this mean?
Critically appraise the information contained within your meta-analysis
What is a good way to assess data (step 6) in your meta-analysis?
Using tables
What is this an example of?
Overall quality assessment of the studies involved in a meta-analysis
Once you have collected all your studies in your meta-analysis, what is the next step for you to take? (step 7)
Calculate the effect sizes
The effect size you will use in your meta-analysis is dependent on…
Your research question
What effect size should you use when examining an association?
Correlation r
What effect sizes measures should you use when comparing two groups with dichotomous outcomes?
risk ratio or odds ratio
What effect sizes measures should you use when comparing two groups with continuous outcomes?
Cohen’s d or Hedges’ g
Step 8 in a meta-analysis is do the meta-analysis
Before you can combine different effect sizes from different studies, you need to look at a number of particular issues.
What are they? (4 in total)
- How were studies with more than one effect size handled?
- Were fixed or random effects models used, and why? (how do you weight the effect sizes across studies)
- How was heterogeneity assessed? (how different were effect sizes in terms of magnitude)
- How was publication bias assessed?
One important aspect of conducting a meta analysis is to ensure data independence
What does this mean?
If one study contibuted more than one effect size this means that the same participants provided multiple outcomes for your review.
The measures for the study are not independent, and it would be inappropriate to treat each effect size as independent
What might you do if a study has more than one effect size (e.g. not data independent)
Average the effect sizes for that study.
Some meta-analysis weight the effect sizes, what does this mean?
They put more weight on the effect size found, depending on the sample size and variance of the study.
Studies with larger study size are given a bigger weight in the overall effect estimate, as they are considered more precise.
When pooling or combining the effect sizes from different studies for a meta-analysis, there are two options you can choose from.
What are they?
important for exam
Fixed or Random model
Fixed effect model: assumes that effect sizes are similar across studies
(homogenous in terms of outcomes, population examined etc.)
Random effect model: asssumes that effect sizes vary from study to study
(effect sizes might be higher or lower when participants are older or more educated than other studies)
What do the following 4 studies assess?
Q statistic
Tau (equivalent to SD of an effect size)
I2 (expressed as a percentage)
Subgroup analyses (note: recommend at least 4 studies per subgroup) and/or meta-regression.
Estimate variablility of effect sizes across studies
i.e Heterogeneity
When assessing heterogeneity of effect sizes in your meta-analysis, what might happen if you have a low number of studies (e.g. 10 or less)
It will have low power
One important issue in a meta-analysis is the assess publication bias, what is it?
important for exam
Refers to the fact that significant findings are more likely to be published in journals than nonsignificant findings
The file-drawer problem is another name for…
publication bias
What are two methods you can use to test publication bias?
Fail-safe N (Nfs)
Funnel plot analysis
What does the fail-safe N (Nfs) tell us?
The statistic represents the number of hypothetical studies that woul dbe required to reduce a given r to a small, non-significant effect
(i.e. r = .10)
What is a funnel plot analysis?
How does it work?
A scatter plot of the effect estimates of individual studies.
These effect estimates are plotted against some measure of each study (e.g. size, or precision, or the standard error of effect estimate)
in the example, each dot represents an effect size from a single study. Studies with larger sample sizes are towards the top and smaller studies at the bottom. In the absence of publication bias, the dots should be relatively symmetrical. When there is publication bias, there are gaps in the funnel (such as the right image).
What is the issue with using a funnel plot for looking at publication bias?
It is visual (very subjective)
- However, there are inbuilt tests in meta-analysis that confirm the results of the funnel plot.*
- If there are not enough studies, the power of the study is small.*
Step 9 of doing a meta analysis is “writing it up”.
What are the features within writing it up?
Tables and/or forest plots to summarise:
o characteristics of the studies (eg. sample ns, country, study
design etc)
o effect size estimates, including p values and CIs
Statement of major findings
Compare/contrast to available research
Critically evalute findings - Alternative explanation for observed results
Generalisability of results (was there bias present)
Limitations of review
Implications for future research, theory, policy, practice
