Ch 19 - Systematic Reviews: Meta-Analysis and Metasynthesis Flashcards
Systematic Research
review that methodically integrates research evidence about a specific research question using careful sampling and data collection procedures that are pre-determined
-disciplined and transparent
QUANTITATIVE
Meta-Analysis Overview
QUANTITATIVE
information from different studies is used to compute a common metric (effect size)
–>effect sizes are averaged across studies, yielding information about the existence of a RELATIONSHIP between variables in many studies, but also the MAGNITUDE
Meta-Analysis Advantages
- Objectivity: draw conclusions, explicit and open to scrutiny, use statistical formulas
- Power: probability of detecting a true relationship between variables, combining effects = power is enhanced
–>but not always appropriate
Criteria for using Meta-Analysis in Systematic Review
- Research question being addressed should be nearly identical across studies (IV, DV, and population)
- Must be sufficient base of knowledge for statistical integration - more than just a few studies included
- Evidence must be consistent - results must not be conflicting
What are the steps in Meta-Analysis?
- Problem formation
- Design
- Literature search
- Evaluation of study quality
- Extraction/encoding of data for analysis
- Calculation of effects
- Data analysis
Meta-Analysis Steps: Problem formation
SR begins with a problem statement and a research question/hypothesis - must be created FIRST
- questions are narrow, focusing
- key constructs are conceptually defined –> critical to deciding whether a primary study qualifies for synthesis
Meta-Analysis Steps: Design
Sampling: primary studies that address research question
- state inclusion and exclusion criteria
- quality is important
- statistical heterogeneity of results - is there variation across studies?
Substantively: criteria discuss key variables and the population
Methodologically: criteria specify parameters (ex. only randomized experimental design will be used)
Practically: choose what to include/exclude (ex. exclude reports in language other than english)
Meta-Analysis Steps: Literature Search
Reviewers must decide whether to cover published/unpublished findings
–>do we use grey literature? - studies with more limited distribution (unpublished)
Publication bias: tendency for published studies to systematically overrepresent statically significant findings (bias against the null hypothesis)
Meta-Analysis Steps: Evaluation of Study Quality
evidence from primary studies needs to be evaluated to assess how much confidence to place in the findings
- may do overall quantitative ratings of evidence quality for each study
- ->domain based evaluation (component approach opposed to scale approach) - individual features are given a rating for each study
- quality of primary studies is judged by 2+ qualified individuals
Meta-Analysis Steps: Extraction and Encoding of Data for Analysis
extract and record relevant information about the findings, methods, and study characteristics of each study in the analysis
- ->Goal: produce a data set amenable to statistical analysis
1. basic source information (year of publication, country where data was collected) is recorded
2. sample size is crucial
3. information about findings must be extracted: calculate effect sizes or record sufficient statistical information that can be computed by a program - ->completed by two or more people to assess interrater agreement
Meta-Analysis Steps: Calculation of Effects
meta-analyses depend on the calculation of an index that encapsulates in a single number the relationship between the IV and DV in each study
Three common groups:
- comparisons of two groups, experimental vs. control (ex. BP)
- comparisons of two groups on a dichotomous outcome (ex. stopped smoking vs. continued smoking)
- correlations between two continuous variables (ex .BP and anxiety scores)
–>seek to get an effect size
Cohen’s d
the effect size index
-transforms all effects into SD units
(ex. d=.50 –> group mean for one group was 1/2 a SD higher than the mean from the other group)
Meta-Analysis Steps: Data Analysis
After an effect size is calculated, a pooled effect estimate is computed as a weighted average of the individual effects
- larger weighted study = larger effect on weighted average
- ->inverse variance method: using standard error to calculate a weight (larger studies = smaller error)
Heterogeneity
- forest plots: visual inspection of heterogeneity
- ->low heterogeneity = use of a fixed effects model, high heterogeneity = use of a random effects model/sensitivity analysis
Study Quality
forest plots
graphs the effect size for each study, together with the 95% confidence interval around each estimate
Sensitivity analysis
refers to an effort to test how sensitive the results of an analysis are to changes in the way the analysis was done
Subgroup analysis
explores moderating effects on effect size –> splitting effect size information into distinct categorical groups (ex. men and women)
How to test study quality?
- establish a quality threshold for study inclusion
- understake sensitive analysis to sdetmerine whether the exclusion of lower-quality studies changes the results of analysis based on the most rigorous studies
- consider quality as the basis for exploring variation in effects (do randomized designs yield different average effect size estimates than quasi-experimental designs?)
- weight studies according to quality criteria (more weight to larger studies)
- often use a mix of strategies
Metasynthesis
QUALITATIVE
a family of methodological approaches to developing new knowledge based on rigorous analysis of existing qualitative research findings
–>product that is more than a sum of it’s parts, offer new insights/interpretations of findings
NOT a literature review or a concept analysis
Steps in a Meta-Synthesis
- problem formulation
- design
- literature search
- evaluation of study quality
- extraction of data for analysis
- data analysis/interpretation
Steps in a Meta-Synthesis: Problem Formulation
Begin with research question or focus for investigation
-key issue: scope of inquiry - must be broad enough to fully capture phenomenon of interest, but focused enough to yield meaningful findings
Steps in a Meta-Synthesis: Design
Requires considerable advance planning - 2+ researchers b/c data is highly subjective
- should involve efforts to enhance integrity and rigor (ex. investigator triangulation)
- make upfront sampling decisions (where to get data)
- using multiple traditions
- excluding studies that are not adequately supported with direct quotes from participants
Steps in a Meta-Synthesis: Literature search
more difficulty to find qualitative studies
Steps in a Meta-Synthesis: Evaluations of study quality
formal evaluations of primary study quality are not as common in meta synthesis as in meta-analysis - must do to at least describe the sample of studies in the review
-may include low and high quality studies - including all that are relevant
Steps in a Meta-Synthesis: Extraction of Data for Analysis
information about various features of each study need to be abstracted and coded as part of the project
- records features of the data source (ex. year of publication), characteristics of the sample (ex. age), and methodologic features (ex. research tradition)
- important info should be extracted: key themes, metaphors, or categories from each study (HARD to do)
Steps in a Meta-Synthesis: Data Analysis/Interpretation
Three approaches:
- Noblit and Hare
- Paterson, Thorne, Canam, and Jillings
- Sandelowski and Barroso
Noblit and Hare Approach to data analysis/interpretation (Meta-ethnography)
argues that integration should be interpretive, not aggregative (not focused on constructed interpretations rather than descriptions)
7 phases of synthesizing qualitative studies
- deciding on phenomenon
- deciding which studies are relevant for synthesis
- reading and rereading each study
- deciding how studies are related
- translating the qualitative studies into one another
- synthesizing translations
Paterson, Thorne, Canam, and Jillings Approach to data analysis/interpretation
Three components:
- Meta-data analysis: study of the results of reported research in a specific area by looking over the “processed data”
- Meta-method: study of the methodoloogic rigor of the studies included in the metasynthesis
- Meta-theory: analysis of the theoretical underpinnings on which the studies are grounded
–>end product is a metasynthesis that results from brining back together the finding of these three meta-study components
Sandelowski and Barroso Approach to data analysis/interpretation
Primary studies = summaries: if they yield descriptive synopses of the qualitative data, lists/frequencies of themes without any conceptual reframing
Syntheses: more interpretive, involve conceptual/metaphorical reframing
Meta-summary: can use both summaries and syntheses
Manifest effect sizes: effect sizes calculated from the manifest content
- frequency effect size: magnitude of findings, number of reports that contain a given finding divided by all other reports
- intensity effect size: concentration of findings within each report, calculated by calculating the number of different findings in a give report divided by the total number of findings all reports
