Lecture 8: Hierarchies of Scientific Evidence Flashcards
What is the hierarchy of evidence?
Meta analysis Systematic review Randomized controlled double blind studies cohort studies case control studies case series case reports ideas, editorials, opinions animal research in vitro research
How to select participants for a study?
o Participants recruited to represent a target population
o Can be randomly selected or selected based on specific exposure status or disease status
What are the possible timings for studies?
o “Prospective” - follows participants and observes exposure/outcome over time
o “Retrospective” – study begins and captures exposure/outcome that has already occurred
o “Cross-sectional”– study captures data at just one point in time
March 20th, 2019
What is a prospective study?
follows participants and observes exposure/outcome over time
What is a retrospective study?
study begins and captures exposure/outcome that has already occurred
What is a cross-sectional study?
study captures data at just one point in time
March 20th, 2019
What are cross sectional studies?
- Observational study of a population (often selected for similar characteristic(s)) at one time point
- Measures prevalence of an outcome
- Collects information on exposure and outcome concurrently
EX: NHANES, CCHS
What are the strengths of cross sectional studies?
Useful for developing hypotheses, discovering novel associations
What are the limitations of cross sectional studies?
Cannot establish temporal sequence between exposure and outcome, or causality.
Enables reporting of statistical associations.
Prevalence vs Incidence?
Prevalence is the proportion of a population who have a specific characteristic in a given time period. Incidence in epidemiology is a measure of the probability of occurrence of a given medical condition in a population within a specified period of time. How many new cases of outcome over a period of time is incidence. The proportion of people who have the outcome at a certain time is prevalence.
What are case control studies?
- sample selection is based on disease status
- cases and controls are usually matched for a number of characteristics (sex, age, presence of other conditions/disease)
What are the strengths of case control studies?
- relatively efficient and inexpensive
- more suitable to study fewer common disease/outcomes
- allows multiple exposures to be studied.
What are the limitations of case-control studies?
- only one outcome can be studied
- temporal sequence difficult to establish
- prone to recall bias
- observational in nature.
What is recall bias in case control studies?
Recall bias –> unique to case control studies. When they don’t really remember what they eat like if you ask a middle ages person what they were eating in high school. This is not exactly recall bias. Recall bias is something about the disease status that is affecting how the cases are recalling information. They might say they had a less healthy diet than they actually did bc they are biased by their disease status. Really difficult to control for this.
What are cohort studies?
observational study that prospectively follows a group over time, some will be exposed to a risk factor and some will not (or high vs low exposure)
involves repeated measures of variables over time in the same individuals (sometimes over decade)
What are the strengths of cohort studies?
- enables study of natural progress of outcome/disease
- multiple outcomes can be incorporated
- temporal sequence is established
- allows measurement of incidence
What are the limitations of cohort studies?
- inefficient for studying rare diseases
- expensive and time-consuming
- difficult to maintain cohort (loss to follow up)
Why are drop-outs problematic in cohort studies?
There might be differences between people who chose to not continue the study and people who stay in the study. When people drop-out of the study, it can bias the data.
What are randomized controlled trials?
- gold standard of clinical trials
- test efficacy of intervention on a population of interest
- placebo compared with a treatment
- causality can be determined
How is randomization beneficial?
randomization improves the design of the study because it controls for confounders which lets you determine causality bc you for sure know that there is no other confounders in this relationship
What is the reporting diagram that should be followed with RCTs?
CONSORT
Consolidated Standards of Reporting Trials
What is randomization?
Evenly distributes exposures of known and unknown factors that might affect study outcome.
What is blinding?
- used to avoid bias in favour of an anticipated effect –> from study participant or researcher who is performing the measurement
- “double blind” indicates that neither participant nor the investigator is aware of whether the participant is receiving the treatment of the placebo
Is blinding possible in nutrition research?
It is possible but very hard to do because if the participant is following a fat free or high fat diet, they can tell the intervention but the researcher or the data analyst can be blinded for it. It is easier to do it for supplementation studies.
What is the hierarchy in reviewing evidence?
- narrative review
- systematic review
- meta-analysis
What is narrative review?
A literature review or narrative review is a type of review article. A literature review is a scholarly paper, which includes the current knowledge including substantive findings, as well as theoretical and methodological contributions to a particular topic.
- potential for bias
- pick and choose articles (there is no systematic way of choosing articles)
What is systematic review?
Systematic reviews are a type of literature review that uses systematic methods to collect secondary data, critically appraise research studies, and synthesize findings qualitatively or quantitatively.
What is meta analysis?
the statistical analysis of a large collection of results from individual literature for the purpose of integrating their respective findings.
What are the two purposes in combining data from independent clinical studies?
- Determine if a similar treatment effect exists for an intervention examined in independent studies. If so, can estimate net effect of the intervention
- If treatment if not similar across studies, can examine factor that may explain differing effects.
What are the limitations overcome by meta-analysis?
- small sample size in clinical trials (type II error)
- increase precision of estimates
- resolve discrepancies
- observe if effects are similar or different across different subgroups (e.g. clinical populations, treatment duration, treatment dose)
True/False
Meta analysis can be done on observational studies
True
Most people think that meta-analysis looks at only intervention studies which will show an effect, but you can do meta-analysis on observational studies and look at associations.
What are the steps in meta-analysis?
- formulate a research question
- define eligibility criteria
- identify studies and extract data
- statistical analysis which include:
- obtain measure of treatment effect for individual studies
- combine studies for a measure of overall effect
- assess study heterogeneity
- assess publication bias
- optional: perform subgroup analysis
- report and interpret results
What is the PRISMA statement?
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
- initially published in 2009, updated in 2015 with new extensions
- PRISMA network is when you are doing a meta-analysis that looks at different interventions for one treatment. Which one is the most effective? You see this a lot in pharmaceutical industry and not in nutrition really?
How to formulate a research question?
PICO/PECO
What does PICO/PECO stand for?
P problem
I/E intervention or Exposure
C comparison
O outcome
Where to register a systematic review protocol?
PROSPERO
What is on the PRISMA checklist?
Title Abstract Introduction Methods Results Discussion Funding
How to measure the treatment effect for individual studies?
Report or compute results from individual studies:
- means (standardized) and standard deviations
- odds ratios or relative risk and confidence intervals
Weights applied to individual studies
- Generic inverse variance method (1/var)
- Larger studies assigned higher weights than smaller studies.
How do you apply weights to individual studies?
- Generic inverse variance method (1/var)
- Larger studies assigned higher weights than smaller studies.
- it is really based on sample size
What are the two types of overall effect when combining studies for measure of overall effect?
fixed effects: assume all studies are evaluating one true underlying effect
random effects: assumes that studies are evaluating underlying effects
both analyses output Z score, p-value, overall RR and 95% CI
What is fixed effects?
assume all studies are evaluating one true underlying effect
What is random effects?
assumes that studies are evaluating different underlying effects
How to assign weights to individual studies with fixed effects?
1/var
How to assign weights to individual studies with random effects?
1/(var + t)
What are the differences in assigning weights to individual studies in random and fixed effects?
Statistically we are doing the same thing, but the random effects model is more conservative. The constant t will be decided by the data analysis program depending on the CI and variabilities of the individual study results.
How to interpret a forest plot?
- each study is individually listed with their number of participants and relative risk (outcome measure or effect size)
- the box is the actual relative risk for these individual studies and the size of the box corresponds to the study weight with largest sample size.
- the whiskers on the boxes represent the confidence interval/variability of the study. If the confidence interval is wide, then that is an indication of imprecision of int he estimate.
- the dashed line in the middle is the null value. Relative risk of 1 means that there is no significant difference. RR>1 adverse association, RR<1 protective effect.
- the diamond is the pooled effect and if it crosses the null then, there is no significant association.
- there is no whiskers on the diamond but the diamond itself is and indication of the confidence interval.
How to determine the heterogeneity across studies?
- look at the forest plot and where each study stands
- Cochran’s Q statistics
- I squared statistics
Explain the Cochran’s Q statistics for heterogeneity?
- chi squared test (p<0.05, reject null)
- test for lack of heterogeneity
- a significant Cochran tells you there is heterogeneity across studies
Explain the I squared statistics for heterogeneity?
- Ranges from 0-100% (<30% mild, 30-50% moderate, >50% substantial)
- quantifies heterogeneity
What type of biases does Cochran’s Risk of Bias Tool look at?
- selection bias
- performance bias
- detection bias
- attrition bias
- reporting bias
What is publication bias?
studies with statistically significant results are more likely to be published
- results of meta-analyses could overstate the effect of an intervention
How to reduce publication bias?
aim to include unpublished data in meta-analyses (difficult to locate)
What are the ways of evaluating publication bias?
funnel plots and egger’s linear regression test
What are funnel plots?
- asymmetric plot suggests studies with negative results are missing from the literature
- asymmetry could be due to other factors (selective reporting)
- the most precise studies are on the top of the triangle
What is Egger’s regression test?
- Plots the precision of the studies (weighted) against the standardized effect
- Regression line originates at the zero point if publication bias is not present (also will get a non-significant p value for beta)
What is subgroup analysis?
pool results in subset of study participants (e.g. disease status, men/women, duration of study)
What are the criticisms of meta analysis?
- comparing apples and oranges?
- quality of meta-analysis only as good as the quality of the individual studies included
- some argue that the most important advantage of a meta-analysis is the ability to assess why different studies produced different results rather than providing an ability to calculate a pooled summary of treatment effect
What were the findings of Tsilas et al. article?
No significant effect between total sugars or fructose in type 2 diabetes but a significant protective effect of sucrose (relates to the purpose of the study was done). The authors commented on this: all sources of sugar in the diet are important to think about. Whole grain, yogurt etc. a lot fo them are sweetened with sucrose and in the results it shows that sucrose has protective effect towards type 2 diabetes. Competing interest, other nutrients in these foods. Reverse causality argument. maybe participants were who have risk of type 2 diabetes so they preferred to avoid these sugars.
What is competing interest?
A competing interest is anything that interferes with, or could reasonably be perceived as interfering with
What is reverse causality argument?
Reverse causation (also called reverse causality) refers either to a direction of cause-and-effect contrary to a common presumption or to a two-way causal relationship
