Secondary Research Flashcards
What is secondary research?
Research conducted using data from other studies
What is recruited instead of patients in secondary studies?
Eligible studies
Types of secondary research
Narrative reviews
Systematic Reviews
Who carries out narrative reviews?
Experts in field of study
Structure of narrative reviews
Experts own opinion supported by selected research evidence
Broad, no specific clinical question
Subjective
Risk of narrative reviews
Prone to bias
Literature not searched methodically
Cannot be replicated due to above
Advantages of narrative reviews
Good introduction to topic
Stimulate interest and controversies
Collate existing data for new hypothesis
Question in narrative reviews
Broad in scope
General
Search methods in narrative reviews
Not usually specified
Comes from experts familiarity with literature
Appraisal of individual studies in narrative reviews
Variable
Usually based on experts opinion
Synthesis of results in narrative reviews
Usually only qualitative summary of studies
Resources in narrative studies
Less time consuming but requires expert in field
What are systematic reviews?
Follow rigorous steps in identifying relevant literature, appraising individual studies and analysing suitable data to synthesise a conclusion
Characteristics of systematic reviews
Focused narrow question
Comprehensive and specified data collection
Uniform criteria for study collection
Quantitative synthesis of data (optional)
What is meta-analysis?
Quantitative synthesis of individual study data in systematic reviews
What is GIGO?
Garbage in, garbage out
What does GIGO refer to?
If primary studies have poor quality, then even if they use sound methodology and statistical procedures, outcome will be meaningless
How is quality of individual trials assessed
Nature of patient sample Outcome studied Length of follow-up Comparability of treatment Methodological factors
What methodological factors are studied to assess quality of individual trials?
Adequacy of sample randomisation
Adequate concealment and control of intervention
Analysis with ITT
Objective, blinded outcome assessment
What should inclusion criteria for studies in systematic reviews consider?
Types of study designs Types of subjects Types of publications Language restrictions Types of interventions Time frame for included studies
What characteristics must a literature review have?
Use of multiple databases
Cross checking of reference list of each study retrieved by direct search
Hand searching for materials unidentified online
Approaching experts to common on any missing studies
Identifying grey literature
What is grey literature?
Unpublished literature e.g. conference abstracts, presentations, posters
What is meta-analysis most often used for?
Assess clinical effectiveness of healthcare interventions
What data does meta-analysis use
Summary data or individual patient data
Why is summary data more often used?
Inability of investigators to supply patient data
Increased costs & time
What does quality of a meta-analysis or systematic review depend on?
Comprehensive literature search
Clearly defined eligibility criteria
Clearly defined & strictly adhered protocol
Predetermined criteria related to quality of trials
Thorough sensitivity analysis
Discussion and analysis of statistical and clinical heterogeneity
Why is thorough sensitivity analysis needed in meta-analysis and systematic reviews?
As assessing quality of study can be difficult since information reported is inadequate for this purpose
Steps for conducting meta-analysis
Literature search
Establish criteria for inclusion and exclusion
Record data from individual studies
Statistical analysis of data
What do meta-analysis use for combining individual trial data?
Weight average of results
What is weighting?
Significance attached to each study based on sample size, precision, external validity and methodological quality
Is weighting dependent on outcome of a study?
No
Modes of analysis in meta-analysis
Fixed effects model
Random effects model
What does fixed effects model assume?
All studies share same common treatment effect
What does random effects model assume?
Studies do not share common treatment efefct
What happens in fixed effect analysis?
Inference restricted to included set of studies
Assumes that only random error within studies can explain observed differences
Ignores between-study variations
When can fixed effect analysis be applied?
If heterogeneity can be safely excluded by testing for it
What happens in random effects analysis?
Assumes that each study shows a different effect which are normally distributed around true mean
This gives proportionally greater weight to smaller studies
Disadvantages of random effects analysis
Susceptible to publication bias
Results in wider less precise confidence intervals
What is used in fixed effect analysis?
Mantel-Haenszel and Peto ratios
When is Mantel-Haenszel ratio useful?
When wide differences exist between studies in ratios of the size of two groups
What type of study designs is Mantel-Haenszel helpful for?
Cohort/case control
What is Peto ratio used for?
RCTs
Why is Peto ratio only used in RCTs?
Can produce bias results in unequal groups
When do both fixed and random effects analysis have similar confidence intervals?
In the absence of heterogeneity
How is heterogeneity calculated?
Q statistic
Advantage of random effects model
Wider confidence interval which may be more representative
Give examples of clinical heterogeneity
Diverse interventions
Differences in selection of patients
Severity of disease
Dose or duration of treatment
Is clinical heterogeneity measurable?
No
What is methodical heterogeneity?
Heterogeneity resulting from differential use of study methodology
Name the types of heterogeneity
Clinical
Methodological
Statistical
What is statistical heterogeneity?
Studies may report same outcome but results that are not consistent with each other
What is a homogenous sample?
Set of studies which have comparable outcomes without much variation
What is a heterogenous sample?
Studies with significant variation amongst them
What can be used to test for statistical heterogeneity?
Forest Plot L'Abbe plot Galbraith plot Chi Square I2 statistic Cochrans Q
What is a L’Abbe plot?
Modified scatter plot where CER is plotted against EER from individual trials
How will data appear in L’Abbe plot?
Trials that report experimental treatment to be superior to control will be in upper left of plot
If experimental is no better than control, point will fall on line of equality
If control is better than experimental point will be in lower right of plot
Axis of a Galbraith plot?
Horizontal: 1/standard error of study effect estimate
Vertical: study effect estimate divided by its standard error (log odds ratio/SE)
What is 1/standard error equal to?
Precision
What is the standard normal deviate?
Study effect estimate divided by its standard error
What can statistical procedures test in meta-analysis re individual studies?
Whether results of study reflect single underlying effect (homogenous) or distribution effect (heterogenous)
Limitation of studying homogeneity/heterogeneity of single studies?
Statistical tests lack power to detect heterogeneity in most meta-analyses
What can Chi square test in meta-analysis?
Test of significance for heterogeneity - does not measure it
How can Chi square test studies in meta analysis for heterogeneity?
Q test
I2 test
What is the rule re Chi square when used to test heterogeneity in meta-analyses?
Its value on average is equal to its degree of freedom, which is n-1 (number of studies -1)
What would Chi Square of 7 for a meta-analysis of 8 studies mean?
N-1 = 7
Chi square = 7
Thus, chi square of 7 or less means no significant heterogeneity
Disadvantages of chi square use for meta-analysis
Can only tell us heterogeneity
Does not tell us anything about homogeneity
Why can Chi square not tell us anything about homogeneity?
Low power
How can one quantify heterogeneity?
Cochrans Q
How does one calculate Cochrans Q?
Weighted sum of squared difference between individual study effects and pooled effect across studies
What does I2 statistic do?
Describes percentage of variation across studies that are due to heterogeneity rather than chance
What does high P mean in chi square tests for meta-analyses?
High P suggests that heterogeneity is insignificant and that one can perform meta-analyses
If there is heterogeneity in meta-analysis, what can one do?
Study reasons behind it
Random effects analysis
What must one consider when studying the reasons behind heterogeneity in meta-analysis?
Clinical differences
Methodological issues e.g. problems with randomisation
Early termination of trials
Use of absolute rather than relative measures of risk etc
Publication bias
What procedures can give suggestions of the causes behind heterogeneity?
Meta-regression analysis
Subgroup analysis
What is publication bias?
Tendency for authors to submit or editors to publish only those studies that yield statistically significant results
What do methods that diagnose presence of publication bias work?
Based on the assumption that small studies are more susceptible to publication bias and may therefore show larger treatment effects
When are methods for publication bias diagnosis not feasible?
When all available studies are equally large i.e. have similar precision
If heterogenous studies
What happens if methods to diagnose publication bias are used in heterogenous studies?
May lead to false positives for publication bias
Methods used to diagnose publication bias
Funnel plots Failsafe N Linear regression Correlation method Cumulative meta-analysis Trim-and-fill procedure
Most common method used to detect publication bias?
Funnel plot
What does a funnel plot show?
Relation between effect size and precision of individual studies in a meta-analysis
Reasoning behind funnel plots?
Small studies are more likely to remain unpublished if their results are not significant and larger studies get published regardless of producing asymmetrical funnel plots.
Axis of funnel plots
X: measure of effect size
Y: measure of precision
Measures of precision
Sample size
Inverse of standard error (1/SE)
What can be used for measures of effect size?
OR
Log OR
Cohens d
Reasoning behind Failsafe N?
Meta-analyses can be wrong due to non-significant studies that are not published.
Inclusion of these studies could nullify the statistical significance of an observed mean effect.
Who developed Failsafe N?
Rosenthal and Cooper
What does Failsafe N do?
Calculates the number of zero-effect studies (no effect) that would be required to nullify the mean effect seen in a meta-analysis
What can be used to quantify publication bias?
Linear regression method Correlation method (Tau)
What can cumulative meta-analysis be used for?
Assess potential impact of publication bias in tilting or nullifying an effect
How are cumulative meta-analysis carried out?
Studies sorted from largest to smallest
Meta-analysis is run with one study, then repeated with 2nd etc
Forest plot/blobbogram plotted; first row will show effect based on one study, 2nd row shows cumulative effect based on 2 studies etc.
Results of cumulative meta-analysis
If the point estimate stabilizes based on the initially added larger studies, then there is no evidence that the smaller studies are producing a biased overall effect.
If point estimate shifts when smaller studies are added, may be bias - and one can also see direction of bias
Who developed the trim-and-fill procedure?
Duval and Tweedie (2000)
What does trim-and-fill procedure do?
Assess whether effect would change if bias were removed
What happens in trim-and-fill procedure?
Type of sensitivity analysis where missing studies are imputed and added to the analysis, and then effect size is recomputed.
What is location bias?
Studies not located due to citation habits, databases used, keywords used or multiple replication of same data
What is inclusion bias?
Reviewers tendency to include studies they agree with
What is sensitivity analysis?
Analysis of the extent to which an outcome derived from research will change when hypothetical data are introduced
What does sensitivity analysis mainly apply to?
Economic analysis
Meta-analysis
What are the nodal points in meta-analysis where hypothetical data could be introduced?
Significant heterogeneity (or absence) - impact of outcome? Factors determining methodological quality Publication bias
Name a sensitivity analysis method where publication bias can be examined
Failsafe N
What is the other name for a forest plot?
Blobbogram
What do forest plots present?
Effect (point estimate) from each individual study as a blob/square with horizontal line across the square
What is the square in forest blot?
Measured effect
What is the horizontal line in forest plot?
9% CI - indicates precision
What does size of square represent in forest plot?
Amount of information in that study - usually the sample size
What does length of horizontal line in forest plot represent?
Degree of uncertainty of estimated treatment effect for study
What is the mid vertical line in forest plots?
Represents null effect for difference in outcomes
What does it mean when a square lies on one side of the vertical line on a forest plot?
Outcome of that study was favoured based on the side of the line the square it is
What will be shown on a forest plot if an individual study result is statistically significant?
Horizontal lines representing uncertainty will lie entirely on the same size as the square and will not cross the vertical line
What does the diamond mean in forest plots?
Final synthesis of individual studies
What does the position of the diamond in forest plots depend on?
Whether the intervention rates favourably or not at the end of the meta-analysis
Things to note in a forest plot
Combined/pooled effect size
CI of individual point estimates and combined estimates
Weighting assigned to studies
Heterogeneity of results
What is the CI of individual point estimates on a forest plot?
Line width
What is the CI of the combined estimate?
Diamond width
What does the size of the squares refer to?
Weighting assigned to study
What would absence of heterogeneity look like on a forest plot?
Vertical linearity rectangles
Why are relative risk and odds ratio more advantageous than absolute risk for calculating effect size in meta-analysis?
Depend less on baseline risk
When are OR and RR appropriate to use as effect size measure?
When both variables are binary
What is conventionally used to show effect size when reporting treatment effects?
Odds ratio
What can occur if one assumes OR and RR are the same?
One may overestimate the effect of treatment - but this overestimation will be small if experimental and control event rates are <20%
At what point will OR be significantly greater than RR?
For frequent events i.e. >20%
Why is relative risk used in Cochrane reviews?
For frequent events, OR will be greater than RR, leading to overestimation of risk
What are the two effect size measures for continuous variables?
Simple difference between the mean values (DM)
Standardized difference between the mean values (SMD)
What is DM?
Keeps original units and is the mean value of one group minus the mean value of the other group
How does one carry out SMD?
Dividing DM by pooled standard deviation of the two groups using a formula
Formula for SMD
SMD = (
What is SMD also known as?
Cohens D
Hedges’ g
What does SMD of 0 mean
Treatment and placebo have same effects
Advantages of Hedges’ g?
Avoids a bias seen in Cohens d when there are small number of subjects
Statistical procedures for fixed effects
Peto
Mantel-Haenszel
Statistical procedures for random effects
DerSimonian-Laired
Measured effects for fixed effects model
OR
Rate Ratios
Risk Ratios
Measured effects of fixed effects model
Approximates OR
Measured effects for random effects model
Ratios and rate differences
What is a Quorum statement?
Consensus statement on standard format for meta-analysis
What is a consort statement?
Consensus statement on standard format for RCTs
What is meta-regression analysis?
Technique of regression wherein regression model is applied to meta-analysis to analyse which characteristics of the studies actually contributed to overall effect size
