Lecture 8: Hierarchies of Scientific Evidence

Question 1

What is the hierarchy of evidence?

Answer 1

Meta analysis
Systematic review
Randomized controlled double blind studies 
cohort studies
case control studies 
case series 
case reports
ideas, editorials, opinions 
animal research 
in vitro research

Question 2

How to select participants for a study?

Answer 2

o Participants recruited to represent a target population

o Can be randomly selected or selected based on specific exposure status or disease status

Question 3

What are the possible timings for studies?

Answer 3

o “Prospective” - follows participants and observes exposure/outcome over time
o “Retrospective” – study begins and captures exposure/outcome that has already occurred
o “Cross-sectional”– study captures data at just one point in time
March 20th, 2019

Question 4

What is a prospective study?

Answer 4

follows participants and observes exposure/outcome over time

Question 5

What is a retrospective study?

Answer 5

study begins and captures exposure/outcome that has already occurred

Question 6

What is a cross-sectional study?

Answer 6

study captures data at just one point in time

March 20th, 2019

Question 7

What are cross sectional studies?

Answer 7

• Observational study of a population (often selected for similar characteristic(s)) at one time point
• Measures prevalence of an outcome
• Collects information on exposure and outcome concurrently

EX: NHANES, CCHS

Question 8

What are the strengths of cross sectional studies?

Answer 8

Useful for developing hypotheses, discovering novel associations

Question 9

What are the limitations of cross sectional studies?

Answer 9

Cannot establish temporal sequence between exposure and outcome, or causality.

Enables reporting of statistical associations.

Question 10

Prevalence vs Incidence?

Answer 10

Prevalence is the proportion of a population who have a specific characteristic in a given time period. Incidence in epidemiology is a measure of the probability of occurrence of a given medical condition in a population within a specified period of time. How many new cases of outcome over a period of time is incidence. The proportion of people who have the outcome at a certain time is prevalence.

Question 11

What are case control studies?

Answer 11

• sample selection is based on disease status
• cases and controls are usually matched for a number of characteristics (sex, age, presence of other conditions/disease)

Question 12

What are the strengths of case control studies?

Answer 12

• relatively efficient and inexpensive
• more suitable to study fewer common disease/outcomes
• allows multiple exposures to be studied.

Question 13

What are the limitations of case-control studies?

Answer 13

• only one outcome can be studied
• temporal sequence difficult to establish
• prone to recall bias
• observational in nature.

Question 14

What is recall bias in case control studies?

Answer 14

Recall bias –> unique to case control studies. When they don’t really remember what they eat like if you ask a middle ages person what they were eating in high school. This is not exactly recall bias. Recall bias is something about the disease status that is affecting how the cases are recalling information. They might say they had a less healthy diet than they actually did bc they are biased by their disease status. Really difficult to control for this.

Question 15

What are cohort studies?

Answer 15

observational study that prospectively follows a group over time, some will be exposed to a risk factor and some will not (or high vs low exposure)

involves repeated measures of variables over time in the same individuals (sometimes over decade)

Question 16

What are the strengths of cohort studies?

Answer 16

• enables study of natural progress of outcome/disease
• multiple outcomes can be incorporated
• temporal sequence is established
• allows measurement of incidence

Question 17

What are the limitations of cohort studies?

Answer 17

• inefficient for studying rare diseases
• expensive and time-consuming
• difficult to maintain cohort (loss to follow up)

Question 18

Why are drop-outs problematic in cohort studies?

Answer 18

There might be differences between people who chose to not continue the study and people who stay in the study. When people drop-out of the study, it can bias the data.

Question 19

What are randomized controlled trials?

Answer 19

• gold standard of clinical trials
• test efficacy of intervention on a population of interest
• placebo compared with a treatment
• causality can be determined

Question 20

How is randomization beneficial?

Answer 20

randomization improves the design of the study because it controls for confounders which lets you determine causality bc you for sure know that there is no other confounders in this relationship

Question 21

What is the reporting diagram that should be followed with RCTs?

Answer 21

CONSORT

Consolidated Standards of Reporting Trials

Question 22

What is randomization?

Answer 22

Evenly distributes exposures of known and unknown factors that might affect study outcome.

Question 23

What is blinding?

Answer 23

• used to avoid bias in favour of an anticipated effect –> from study participant or researcher who is performing the measurement
• “double blind” indicates that neither participant nor the investigator is aware of whether the participant is receiving the treatment of the placebo

Question 24

Is blinding possible in nutrition research?

Answer 24

It is possible but very hard to do because if the participant is following a fat free or high fat diet, they can tell the intervention but the researcher or the data analyst can be blinded for it. It is easier to do it for supplementation studies.

Question 25

What is the hierarchy in reviewing evidence?

Answer 25

- narrative review - systematic review - meta-analysis

Question 26

What is narrative review?

Answer 26

A literature review or narrative review is a type of review article. A literature review is a scholarly paper, which includes the current knowledge including substantive findings, as well as theoretical and methodological contributions to a particular topic. - potential for bias - pick and choose articles (there is no systematic way of choosing articles)

Question 27

What is systematic review?

Answer 27

Systematic reviews are a type of literature review that uses systematic methods to collect secondary data, critically appraise research studies, and synthesize findings qualitatively or quantitatively.

Question 28

What is meta analysis?

Answer 28

the statistical analysis of a large collection of results from individual literature for the purpose of integrating their respective findings.

Question 29

What are the two purposes in combining data from independent clinical studies?

Answer 29

1. Determine if a similar treatment effect exists for an intervention examined in independent studies. If so, can estimate net effect of the intervention 2. If treatment if not similar across studies, can examine factor that may explain differing effects.

Question 30

What are the limitations overcome by meta-analysis?

Answer 30

- small sample size in clinical trials (type II error) - increase precision of estimates - resolve discrepancies - observe if effects are similar or different across different subgroups (e.g. clinical populations, treatment duration, treatment dose)

Question 31

True/False Meta analysis can be done on observational studies

Answer 31

True Most people think that meta-analysis looks at only intervention studies which will show an effect, but you can do meta-analysis on observational studies and look at associations.

Question 32

What are the steps in meta-analysis?

Answer 32

- formulate a research question - define eligibility criteria - identify studies and extract data - statistical analysis which include: - obtain measure of treatment effect for individual studies - combine studies for a measure of overall effect - assess study heterogeneity - assess publication bias - optional: perform subgroup analysis - report and interpret results

Question 33

What is the PRISMA statement?

Answer 33

Preferred Reporting Items for Systematic Reviews and Meta-Analyses - initially published in 2009, updated in 2015 with new extensions - PRISMA network is when you are doing a meta-analysis that looks at different interventions for one treatment. Which one is the most effective? You see this a lot in pharmaceutical industry and not in nutrition really?

Question 34

How to formulate a research question?

Answer 34

PICO/PECO

Question 35

What does PICO/PECO stand for?

Answer 35

P problem I/E intervention or Exposure C comparison O outcome

Question 36

Where to register a systematic review protocol?

Answer 36

PROSPERO

Question 37

What is on the PRISMA checklist?

Answer 37

``` Title Abstract Introduction Methods Results Discussion Funding ```

Question 38

How to measure the treatment effect for individual studies?

Answer 38

Report or compute results from individual studies: - means (standardized) and standard deviations - odds ratios or relative risk and confidence intervals Weights applied to individual studies - Generic inverse variance method (1/var) - Larger studies assigned higher weights than smaller studies.

Question 39

How do you apply weights to individual studies?

Answer 39

- Generic inverse variance method (1/var) - Larger studies assigned higher weights than smaller studies. - it is really based on sample size

Question 40

What are the two types of overall effect when combining studies for measure of overall effect?

Answer 40

fixed effects: assume all studies are evaluating one true underlying effect random effects: assumes that studies are evaluating underlying effects both analyses output Z score, p-value, overall RR and 95% CI

Question 41

What is fixed effects?

Answer 41

assume all studies are evaluating one true underlying effect

Question 42

What is random effects?

Answer 42

assumes that studies are evaluating different underlying effects

Question 43

How to assign weights to individual studies with fixed effects?

Answer 43

1/var

Question 44

How to assign weights to individual studies with random effects?

Answer 44

1/(var + t)

Question 45

What are the differences in assigning weights to individual studies in random and fixed effects?

Answer 45

Statistically we are doing the same thing, but the random effects model is more conservative. The constant t will be decided by the data analysis program depending on the CI and variabilities of the individual study results.

Question 46

How to interpret a forest plot?

Answer 46

- each study is individually listed with their number of participants and relative risk (outcome measure or effect size) - the box is the actual relative risk for these individual studies and the size of the box corresponds to the study weight with largest sample size. - the whiskers on the boxes represent the confidence interval/variability of the study. If the confidence interval is wide, then that is an indication of imprecision of int he estimate. - the dashed line in the middle is the null value. Relative risk of 1 means that there is no significant difference. RR>1 adverse association, RR<1 protective effect. - the diamond is the pooled effect and if it crosses the null then, there is no significant association. - there is no whiskers on the diamond but the diamond itself is and indication of the confidence interval.

Question 47

How to determine the heterogeneity across studies?

Answer 47

- look at the forest plot and where each study stands - Cochran's Q statistics - I squared statistics

Question 48

Explain the Cochran's Q statistics for heterogeneity?

Answer 48

- chi squared test (p<0.05, reject null) - test for lack of heterogeneity - a significant Cochran tells you there is heterogeneity across studies

Question 49

Explain the I squared statistics for heterogeneity?

Answer 49

- Ranges from 0-100% (<30% mild, 30-50% moderate, >50% substantial) - quantifies heterogeneity

Question 50

What type of biases does Cochran's Risk of Bias Tool look at?

Answer 50

- selection bias - performance bias - detection bias - attrition bias - reporting bias

Question 51

What is publication bias?

Answer 51

studies with statistically significant results are more likely to be published - results of meta-analyses could overstate the effect of an intervention

Question 52

How to reduce publication bias?

Answer 52

aim to include unpublished data in meta-analyses (difficult to locate)

Question 53

What are the ways of evaluating publication bias?

Answer 53

funnel plots and egger's linear regression test

Question 54

What are funnel plots?

Answer 54

- asymmetric plot suggests studies with negative results are missing from the literature - asymmetry could be due to other factors (selective reporting) - the most precise studies are on the top of the triangle

Question 55

What is Egger's regression test?

Answer 55

- Plots the precision of the studies (weighted) against the standardized effect - Regression line originates at the zero point if publication bias is not present (also will get a non-significant p value for beta)

Question 56

What is subgroup analysis?

Answer 56

pool results in subset of study participants (e.g. disease status, men/women, duration of study)

Question 57

What are the criticisms of meta analysis?

Answer 57

- comparing apples and oranges? - quality of meta-analysis only as good as the quality of the individual studies included - some argue that the most important advantage of a meta-analysis is the ability to assess why different studies produced different results rather than providing an ability to calculate a pooled summary of treatment effect

Question 58

What were the findings of Tsilas et al. article?

Answer 58

No significant effect between total sugars or fructose in type 2 diabetes but a significant protective effect of sucrose (relates to the purpose of the study was done). The authors commented on this: all sources of sugar in the diet are important to think about. Whole grain, yogurt etc. a lot fo them are sweetened with sucrose and in the results it shows that sucrose has protective effect towards type 2 diabetes. Competing interest, other nutrients in these foods. Reverse causality argument. maybe participants were who have risk of type 2 diabetes so they preferred to avoid these sugars.

Question 59

What is competing interest?

Answer 59

A competing interest is anything that interferes with, or could reasonably be perceived as interfering with

Question 60

What is reverse causality argument?

Answer 60

Reverse causation (also called reverse causality) refers either to a direction of cause-and-effect contrary to a common presumption or to a two-way causal relationship