7 - Systematic Reviews

Question 1

Define systematic review. Does it include meta analysis?

Answer 1

• application of scientific (or systematic) strategies to limit bias of relevant studies in a specific topic (ie in the gathering, critical appraisal, and synthesis)
• may or may not include meta analysis

Question 2

when is it appropriate to do a systematic review vs meta analysis?

Answer 2

• it is always appropriate to do a systematic review, but not always appropriate to pool data (as in a meta analysis) - ie it may be misleading

Question 3

define a meta-analysis

Answer 3

• a statistical analysis of results from independent studies (used to produce a single estimate of the treatment effect)
• aka pooling (averaging results together)
• generally weighted by error or sample size within that study - ie smaller study has less of an impact on the average from a larger study

Question 4

describe a forest plot. size of CIs wrt box size

Answer 4

• represents meta analysis
• on the left list studies in order of date
• plot square size representing the size of the study
• larger box size = larger studies = smaller CIs
• diamond at the bottom represents pooled estimate of effect (either by error or sample size)

Question 5

what are some reasons to do systematic reviews?

Answer 5

• single studies often do not find significant effects (too small n size, pool for more)
• we can answer questions about subgroups
• can extend the generalizability of results

Question 6

what is involved in a systematic review?

Answer 6

• formulate the question(s)
• determine criteria for inclusion/exclusion of studies
• conduct literature search
• select relevant studies
• assess quality of included studies
• extract the data
• assess sources of heterogenity
• analyze and present results

Question 7

name some criteria for eligibility of studies in SR

Answer 7

• study design, population, intervention, outcome, follow-up length, methodological quality (leave this part out at the beginning though!)

Question 8

main points about search strategies

Answer 8

• use more than one source (there are peer-reviewed and unpublished sources - ie cochrine controlled trials register, clinicaltrials.gov)

Question 9

main points about selection of relevant studies (pt 1 and pt 2)

Answer 9

Part 1
- pairs of reviewers search for titles and abstracts independently to identify articles that should be reviewed in full text
- first need to define eligibility criteria, then create/pilot the data form, independently review (as exclude or include/review full text), determine/report agreement, unweighted Kappa
Part 2
- expand and further define eligibility criteria
- same steps as before but with full text screening (include/exclude/uncertain), keep data of excluded studies and provide a reason for it, kappa again

Question 10

what is kappa/how do you calculate kappa?

Answer 10

• kappa lets the readers know how well we agreed - greater value = more agreed = well defined search = more likely to include important studies
• if not to questions, exclude

Question 11

describe how to determine completeness of the systematic review search you did

Answer 11

• look at cited works in articles you included
• look back and basically determine whether you did a good job or not
• look for evidence of publication bias

Question 12

what is publication bias?

Answer 12

• omission of studies that should be included
• when positive trials are more likely to be published than negative trials it is a publication bias
• when studies are omitted (bc they were never published or maybe we didn’t do a good job in our search)

Question 13

what is time-lag bias?

Answer 13

• when positive trials are more likely to be published rapidly

Question 14

what is a language bias?

Answer 14

when positive trials are more likely to be published in English

Question 15

what is a multiple (duplication) publication bias?

Answer 15

when a positive trial is more likely to be published more than once

Question 16

what is a citation bias?

Answer 16

when a positive trial is more likely to be cited by others

Question 17

what does exclusion of small negative studies do to estimation of effect?

Answer 17

it tends to overestimate the estimation effect

Question 18

what is the most common graphical method to detect publication bias?

Answer 18

funnel plot - plots treatment effect by trial size/standard error

Question 19

what is effect size again? what are small medium large values?

Answer 19

• can be calculated a number of different ways
• basically a simple way of quantifying the difference between two groups without confounding this with n-size (better than p-values alone)
• 1 = large, 0.5 = medium, 0.2 = small

Question 20

Look at the funnel plots and describe where small negative studies are, compared to larger studies (also label axes)

Answer 20

• y axis = standard error, x is odds ratio or effect size
• small negative studies (right and low)
• larger studies higher on y axis (has to do with effect size)
• treatment effect (expressed n OR, a type of treatment effect) are larger for small studies
• if skewed to the left, missing the negative small studies

Question 21

Describe the trim and fill method for funnel plots

Answer 21

trim = take top part of the triangle that is complete and calculate OR
fill = statistically fill in triangle and calculate that OR

Question 22

What is quality assessment? Describe. How is external validity addressed?

Answer 22

• indicates the internal validity of each included study
• evidence shows differences in treatment effect for high-quality vs low quality studies (ie low quality bias in favour of treatment)
• external validity would be how you pick which studies to include/how you look at heterogenity

Question 23

Why do people like using quality scales?

Answer 23

• avoids thinking (maybe they don’t know how to identify important individual criteria)
• easier to portray than assessing each individual criteria
• exclude studies of low quality
• weight studies according to their quality rating
• explain heterogenity

Question 24

what are concerns about the validity of quality scales?

Answer 24

• scoring implies weighting of criteria (do we give more important criteria more weight? did the study give the most important criteria the most weight?)
• scales often include external validity or reporting criteria (should not be included)

Question 25

describe the importance of individual criteria for including studies in SR

Answer 25

• it is dependent on the influence on internal validity for a specific trial type (ie concealment is more important for RCTs etc)

Question 26

Describe the results of the Juni et al. study

Answer 26

• applied 25 quality scales to studies
• found wide range of effect size
• some cases showed no correlation btw quality score and effect size, others showed counter-intuitive results - check out graph (p 10)

Question 27

Describe the Jahad scale

Answer 27

• look at p 11
• the scale is not perfect (includes some things it should and some it shouldn’t, missing some things too) - ie reporting issues should not be mentioned here
• want to know in terms of internal validity!
• look at second example too

Question 28

Describe rating quality - what you should do for a quality assessment (determining included studies)

Answer 28

• consider blinding
• create and pilot forms
• duplicate independent ratings (ie things in partners)
• discussion and consensus
• determine reporting agreement (lets readers know how well everyone understood what was going on)

Question 29

In terms of determining reported agreement, what types of analysis are there?

Answer 29

• ICC if quality rating conclusion is a score
• weighted kappa if QR is an order category
• kappa if QR is a non-order category

Question 30

describe data extraction for included studies

Answer 30

• consider blinding data extractors to author, institution, journal etc
• create and pilot forms
• duplicate independent review (ie things in partners)
• do you need to write to author for more info?
• discussion of results w consensus (ie how do we deal w disagreements - needs to be transparent for readers)

Question 31

describe the cochrane collaboration tool for assessing risk of bias

Answer 31

• pg 12

Question 32

What is a meta-analysis (pooled analysis) and when is it appropriate?

Answer 32

• estimates a single treatment effect (therapy) or sensitivity/specificity (diagnostic), estimates of prognosis etc
• if results are not heterogeneous/heterogeneity can be explained (if not, do NOT pool results)
• note that explanations for heterogeneity must be done a priori! (if the hypothesis we came up with doesn’t explain the heterogeneity we should not pool results!)
• see example on page 13

Question 33

what does heterogenous mean?

Answer 33

• that the results of the included studies are variable

Question 34

what are potential sources of heterogeneity? external or internal validity? examples.

Answer 34

- clinical heterogeneity (external) = difference in clinically important features (patient selection, baseline disease severity, administration of intervention, management of outcomes, adverse events, complications, duration of follow ups etc)
methodological heterogeneity (internal) = refers to differences in methodology (randomization methods, allocation concealment, proportion lost to follow up etc)

Question 35

what is directionality?

Answer 35

• saying that something has a larger or smaller effect and not just that there is a difference in treatment
• always include directionality when possible

Question 36

list 2 ways to evaluate heterogeneity

Answer 36

cochrane chi-square test and I^2 value

Question 37

what is the cochrane chi-square test

Answer 37

• a heterogeneity test
• tests whether estimates of effect between studies are similar (homogonous), low power, p<0.05 implies significant heterogeneity (bad)
• this is one case where we want a larger p!
• effected by amount of studies in your review (small = not as much heterogeneity found, large may find heterogeneity when there really isnt)

Question 38

what is the I^2 value?

Answer 38

• estimates the percentage of total variation that is due to heterogeneity among studies rather than due to chance
• low (25%), medium (50%), high (75%)
• so smaller value is better (means small part of variation is due to heterogeneity or differences btw studies)

Question 39

list the steps for evaluating heterogeneity

Answer 39

1. create hypothesis that can be tested later if heterogeneity is found (a priori!!) - eg studies that included only patients w severe disease will show smaller treatment effects than studies that only included patients w mild/moderate disease
2. conduct the analysis and create forest plot
3. test your hypothesis (start by separating studies into those that included only severe patients and mild/mod patients)
4. re-run analysis and produce new forest plots - see if heterogeneity went away! - if yes, this explains it
5. present the results for each group (conclusions/recommendations should be separate for each group)

Question 40

describe within study variability - fixed/random effects?

Answer 40

• for conducting meta-analysis
• variability if the same study with the same subjects is repeated
• fixed and random effects

Question 41

describe between-study variability - fixed/random effects?

Answer 41

• variability if the same study is repeated in a different population
• only random effects

Question 42

explain random vs fixed effects - which is more conservative?

Answer 42

• how we are going to pool the results for a meta analysis (remember signal over noise)
• for random, the denominator includes more sources of noise, therefore tougher to see the signal (this is more conservative so we want this!)
• for fixed effects, does not include within study variability - can see signal more clearly (its assuming that all studies pool from same population) so less conservative

Question 43

describe choice of summary measures for meta-analysis (weighted mean diff or standardized mean diff)

Answer 43

• WMD (OR, RR, mean), means that all the studies pooled into the systematic review used the exact same means to measure outcome (rare to have this)
• SMD (standard deviation units) means you have to place all the scores on a standardized or z-scale and then combine them

Question 44

how can you use the results of a meta analysis? as a clinician?

Answer 44

• clinical decision-making guidelines
• policy making
• ethical design of future trials
• can improve our generalizability, gives us more applicability, more precision around estimate of effect, can say we are now more certain about these results
• as clinician remember: variation is likely quantitative, not qualitative, can consider results in most reverent subgroups, still requires incorporation of patient preferences (this bullet not talked about much in class!)