Ch 19 - Systematic Reviews: Meta-Analysis and Metasynthesis

Question 1

Systematic Research

Answer 1

review that methodically integrates research evidence about a specific research question using careful sampling and data collection procedures that are pre-determined
-disciplined and transparent
QUANTITATIVE

Question 2

Meta-Analysis Overview

Answer 2

QUANTITATIVE
information from different studies is used to compute a common metric (effect size)
–>effect sizes are averaged across studies, yielding information about the existence of a RELATIONSHIP between variables in many studies, but also the MAGNITUDE

Question 3

Meta-Analysis Advantages

Answer 3

1. Objectivity: draw conclusions, explicit and open to scrutiny, use statistical formulas
2. Power: probability of detecting a true relationship between variables, combining effects = power is enhanced

–>but not always appropriate

Question 4

Criteria for using Meta-Analysis in Systematic Review

Answer 4

1. Research question being addressed should be nearly identical across studies (IV, DV, and population)
2. Must be sufficient base of knowledge for statistical integration - more than just a few studies included
3. Evidence must be consistent - results must not be conflicting

Question 5

What are the steps in Meta-Analysis?

Answer 5

1. Problem formation
2. Design
3. Literature search
4. Evaluation of study quality
5. Extraction/encoding of data for analysis
6. Calculation of effects
7. Data analysis

Question 6

Meta-Analysis Steps: Problem formation

Answer 6

SR begins with a problem statement and a research question/hypothesis - must be created FIRST

• questions are narrow, focusing
• key constructs are conceptually defined –> critical to deciding whether a primary study qualifies for synthesis

Question 7

Meta-Analysis Steps: Design

Answer 7

Sampling: primary studies that address research question

• state inclusion and exclusion criteria
• quality is important
• statistical heterogeneity of results - is there variation across studies?

Substantively: criteria discuss key variables and the population

Methodologically: criteria specify parameters (ex. only randomized experimental design will be used)

Practically: choose what to include/exclude (ex. exclude reports in language other than english)

Question 8

Meta-Analysis Steps: Literature Search

Answer 8

Reviewers must decide whether to cover published/unpublished findings
–>do we use grey literature? - studies with more limited distribution (unpublished)

Publication bias: tendency for published studies to systematically overrepresent statically significant findings (bias against the null hypothesis)

Question 9

Meta-Analysis Steps: Evaluation of Study Quality

Answer 9

evidence from primary studies needs to be evaluated to assess how much confidence to place in the findings

• may do overall quantitative ratings of evidence quality for each study
• ->domain based evaluation (component approach opposed to scale approach) - individual features are given a rating for each study
• quality of primary studies is judged by 2+ qualified individuals

Question 10

Meta-Analysis Steps: Extraction and Encoding of Data for Analysis

Answer 10

extract and record relevant information about the findings, methods, and study characteristics of each study in the analysis

• ->Goal: produce a data set amenable to statistical analysis
  1. basic source information (year of publication, country where data was collected) is recorded
  2. sample size is crucial
  3. information about findings must be extracted: calculate effect sizes or record sufficient statistical information that can be computed by a program
• ->completed by two or more people to assess interrater agreement

Question 11

Meta-Analysis Steps: Calculation of Effects

Answer 11

meta-analyses depend on the calculation of an index that encapsulates in a single number the relationship between the IV and DV in each study

Three common groups:

1. comparisons of two groups, experimental vs. control (ex. BP)
2. comparisons of two groups on a dichotomous outcome (ex. stopped smoking vs. continued smoking)
3. correlations between two continuous variables (ex .BP and anxiety scores)

–>seek to get an effect size

Question 12

Cohen’s d

Answer 12

the effect size index
-transforms all effects into SD units

(ex. d=.50 –> group mean for one group was 1/2 a SD higher than the mean from the other group)

Question 13

Meta-Analysis Steps: Data Analysis

Answer 13

After an effect size is calculated, a pooled effect estimate is computed as a weighted average of the individual effects

• larger weighted study = larger effect on weighted average
• ->inverse variance method: using standard error to calculate a weight (larger studies = smaller error)

Heterogeneity

• forest plots: visual inspection of heterogeneity
• ->low heterogeneity = use of a fixed effects model, high heterogeneity = use of a random effects model/sensitivity analysis

Study Quality

Question 14

forest plots

Answer 14

graphs the effect size for each study, together with the 95% confidence interval around each estimate

Question 15

Sensitivity analysis

Answer 15

refers to an effort to test how sensitive the results of an analysis are to changes in the way the analysis was done

Question 16

Subgroup analysis

Answer 16

explores moderating effects on effect size –> splitting effect size information into distinct categorical groups (ex. men and women)

Question 17

How to test study quality?

Answer 17

• establish a quality threshold for study inclusion
• understake sensitive analysis to sdetmerine whether the exclusion of lower-quality studies changes the results of analysis based on the most rigorous studies
• consider quality as the basis for exploring variation in effects (do randomized designs yield different average effect size estimates than quasi-experimental designs?)
• weight studies according to quality criteria (more weight to larger studies)
• often use a mix of strategies

Question 18

Metasynthesis

Answer 18

QUALITATIVE
a family of methodological approaches to developing new knowledge based on rigorous analysis of existing qualitative research findings
–>product that is more than a sum of it’s parts, offer new insights/interpretations of findings

NOT a literature review or a concept analysis

Question 19

Steps in a Meta-Synthesis

Answer 19

1. problem formulation
2. design
3. literature search
4. evaluation of study quality
5. extraction of data for analysis
6. data analysis/interpretation

Question 20

Steps in a Meta-Synthesis: Problem Formulation

Answer 20

Begin with research question or focus for investigation
-key issue: scope of inquiry - must be broad enough to fully capture phenomenon of interest, but focused enough to yield meaningful findings

Question 21

Steps in a Meta-Synthesis: Design

Answer 21

Requires considerable advance planning - 2+ researchers b/c data is highly subjective

• should involve efforts to enhance integrity and rigor (ex. investigator triangulation)
• make upfront sampling decisions (where to get data)
• using multiple traditions
• excluding studies that are not adequately supported with direct quotes from participants

Question 22

Steps in a Meta-Synthesis: Literature search

Answer 22

more difficulty to find qualitative studies

Question 23

Steps in a Meta-Synthesis: Evaluations of study quality

Answer 23

formal evaluations of primary study quality are not as common in meta synthesis as in meta-analysis - must do to at least describe the sample of studies in the review
-may include low and high quality studies - including all that are relevant

Question 24

Steps in a Meta-Synthesis: Extraction of Data for Analysis

Answer 24

information about various features of each study need to be abstracted and coded as part of the project

• records features of the data source (ex. year of publication), characteristics of the sample (ex. age), and methodologic features (ex. research tradition)
• important info should be extracted: key themes, metaphors, or categories from each study (HARD to do)

Question 25

Steps in a Meta-Synthesis: Data Analysis/Interpretation

Answer 25

Three approaches:

1. Noblit and Hare
2. Paterson, Thorne, Canam, and Jillings
3. Sandelowski and Barroso

Question 26

Noblit and Hare Approach to data analysis/interpretation (Meta-ethnography)

Answer 26

argues that integration should be interpretive, not aggregative (not focused on constructed interpretations rather than descriptions)

7 phases of synthesizing qualitative studies

1. deciding on phenomenon
2. deciding which studies are relevant for synthesis
3. reading and rereading each study
4. deciding how studies are related
5. translating the qualitative studies into one another
6. synthesizing translations

Question 27

Paterson, Thorne, Canam, and Jillings Approach to data analysis/interpretation

Answer 27

Three components:

1. Meta-data analysis: study of the results of reported research in a specific area by looking over the “processed data”
2. Meta-method: study of the methodoloogic rigor of the studies included in the metasynthesis
3. Meta-theory: analysis of the theoretical underpinnings on which the studies are grounded

–>end product is a metasynthesis that results from brining back together the finding of these three meta-study components

Question 28

Sandelowski and Barroso Approach to data analysis/interpretation

Answer 28

Primary studies = summaries: if they yield descriptive synopses of the qualitative data, lists/frequencies of themes without any conceptual reframing

Syntheses: more interpretive, involve conceptual/metaphorical reframing

Meta-summary: can use both summaries and syntheses

Manifest effect sizes: effect sizes calculated from the manifest content

1. frequency effect size: magnitude of findings, number of reports that contain a given finding divided by all other reports
2. intensity effect size: concentration of findings within each report, calculated by calculating the number of different findings in a give report divided by the total number of findings all reports