Post-quiz 4 content Flashcards
Compare the purpose of examining internal and external validity.
External → will the results generalize to the population?
Internal → was the outcome due to exposure, or something else?
What are threats to internal validity?
Name 7.
Factors that can interfere with findings from experimental studies.
- History
- Maturation
- Testing
- Selection
- Instrumentation
- Regression to the mean
- Experimental mortality/attrition
What type of study design is this?
Pre-test, post-test quasi experimental study
(i.e., no randomization or controls)
Describe history as a threat to internal validity.
- Specific events that occurred during the study period.
- External factors that influencing outcome that were not part of the intervention.
- E.g., During the intervention, the school cafeteria started serving more fruits and vegetables.
Describe maturation as a threat to internal validity.
- A function of the passage of time between pre-test and post-test (i.e., people change/grow over time)
- E.g., during the intervention, the students got older and became less picky eaters (they started to like fruits and veggies)
Describe effects of testing as a threat to internal validity.
- Pretesting affects performance on later measurements (post-test)
- E.g., After doing the baseline fruit & veg assessment, the students became more aware of the importance of fruits and veggies and started eating ore of them.
What study design should be used if you were concerned about the effects of testing?
Solomon 4-group
Which threats to internal validity can be reduced by using concurrent groups?
History
Maturation
Testing
What notation correctly describes the study?
A
Describe selection as a threat to internal validity.
- Study groups are not comparable/equivalent
- Selection-history
- Selection-maturation
- Selection-testing
- E.g., schools who received the intervention also have free fruit and veg basket that students can pick from; controls did not have this.
How can selection as a threat to internal validity be controlled?
- Random assignment → if randomization is successful, groups should be similar at baseline.
Describe instrumentation as a threat to internal validity.
- Variability in the analysis and evaluation of the outcomes
- Analytical instruments not calibrated or differently calibrated between experiments.
- Two graders scoring pre/post-tests differently
- E.g., FFQ not validated for study population.
- E.g., 2: Raters assessing post study diet after the nutrition intervention overestimated F&V intake
How can instrumentation as a threat to validity be controlled?
By ensuring measuring instruments are reliable.
(e.g., test-retest, parallel forms, inter/intra rater reliability testing → ensure tools are reliable and give same responses each time to minimize error)
Describe regression to the mean as a threat to internal validity.
- If first values measured are extreme, the post-test values may ‘regress to the mean’ simply as a result of day-to-day variation.
- E.g., by ‘chance’ class A scored very low on pre-test (extreme finding). On second testing, their scores moved towards the mean → may conclude class A improved more when this is actually just ‘regression to the mean’
Describe experimental mortality/attrition as a threat to internal validity.
- Loss to follow-up
- A concern if characteristics of dropouts differ from those who remain in the study, or if dropout rates differ between intervention and control group.
What is intention to treat?
‘once randomized, always analyzed’
Participants are included in analysis in the group they were originally assigned to regardless of whether they completed the trial.
What study design best describes the following:
- 8 elementary schools assigned to receive intervention or not (controls)
- Intervention designed to improve diets, increase physical activity, and decrease screen time
- Assessed students’ diets before and after the intervention
Pre-test, post-test, quasi experimental parallel study
At baseline, the groups did not have different fruit and vegetable intakes.
What threat to internal validity exists here?
Selection
What needs to be considered for external validity?
Need to consider how findings from a study can be generalized to the target population in terms of:
- Sample - representative of population?
- Setting - realistic?
- Study conditions - impact results?
How can external validity be increased?
- Random selection of participants from a population → increase representativeness and ability to generalize results
- Account for the research context → specific aspects of the study may limit the generalizability of the results outside the research settings.
What are advantages of randomized controlled trials? [5]
- Controls for possible confounders → aims to have comparable groups by randomization
- Level of exposure controlled/manipulated by researcher
- Control group for comparison (ensure effects are due to treatment)
- If blinded, reduces bias
- Provides evidence about temporality of associations
What are limits to randomized controlled trials?
- Cost
- Feasibility → ethical implications
- Large sample size needed for statistical power
- Poor compliance and drop outs
- External validity/generalizability
What are 5 nutrition RCT challenges?
-
Duration
- Longer interventions → difficult to have compliance with dietary interventions over long term
- Shorter interventions → long enough to have impact?
-
Controlling exposure
- Difficult to completely ‘control’ intake of a nutrient
- Difficult to have good compliance with nutrition interventions
-
Level of exposure
- Cannot have zero intake group of essential nutrient
- Typically comparing ‘some’ intake with ‘more’ → how to define ‘some’ and ‘more’; ‘some’ may already be enough for outcome of interest → threshold effects
-
Nutrients vs. food
- Will supplement of omega-3 fatty acid produce the same effect as obtaining omega-3 fatty acid from eating fish?
-
Difficult to accurately measure nutrient intake
- Rely on memory and subject’s ability to accurately recall foods consumed & portion sizes
- Foods consumed vary meal-to-meal and day-to-day
- Similar foods may vary in nutrient content.
Thinking critically…
Do omega-3 fats actually have no risk on CVD factors? OR… [5]
- Did subjects already have adequate intakes at baseline?
- Was dose provided clinically relevant?
- Were results influenced by the use of concurrent treatments (e.g., statins)
- Are omega-3 fats only beneficial for prevention of primary CVD not secondary CVD?
- Could effects of omega-3 on CVD risk factors (from observational studies) be explained by something else? (e.g., fish intake, vitamin D and protein, etc.)
Review papers are sometimes called:
Secondary research
What is a narrative review?
What is a problem with narrative reviews?
- ‘expert’ in a field writes article that summarizes the evidence
- can reflect the state of the field, be a summary of current/past research on a topic, provide new opinions, suggest new hypotheses or areas for future research.
- Very useful if you have no idea where to start!
- PROBLEM → High risk of bias (the expert selects the studies included in the review and can focus on those that support his/her opinion)
What is a systematic review?
- Summarizes research on a specific question
- Explicit methods are used to (1) search, (2) appraise, and (3) synthesize the literature
- Potential for bias reduced → all articles meeting the inclusion criteria should be considered.
Rank narrative reviews, systematic reviews and meta-analyses, and RCTs in the hierarchy of evidence for clinical decision making.
Systematic reviews and meta-analyses
RCTs
Narrative reviews
Rank narrative reviews, systematic reviews and meta-analyses, and RCTs in the hierarchy of evidence for clinical decision making.
Systematic reviews and meta-analyses
RCTs
Narrative reviews
What are meta-analyses?
- Used statistical analysis to calculate a summary estimate based on the results of published studies.
- Provides an average of the published results, weighted for differences in the quality/size of the studies
- Usually based on systematic reviews.
What are the 8 steps in systematic review methodology?
- Define research question
- Outline eligibility criteria
- Comprehensively search the literature
- Assess studies and determine which to include
- Assemble a comprehensive data set of studies
- Assess quality of studies
- Perform meta-analysis if appropriate
- Answer research question.
How is a research question defined?
PICO
population
intervention
control/comparison
outcome
Discuss outline eligibility criteria for systematic reviews.
- Studies are ‘subjects’
- What types of studies:
- Only RCTs → all study designs?
- Participants → what age/sex/ethnicity/health status?
- Year of publication → any year, last 5 years?
- What outcomes?
Discuss outline eligibility criteria for systematic reviews.
- Studies are ‘subjects’
- What types of studies:
- Only RCTs → all study designs?
- Participants → what age/sex/ethnicity/health status?
- Year of publication → any year, last 5 years?
- What outcomes?
Which of the following strategies would ensure you get the most papers in your search?
Combining MeSH and keywords with OR
How is literature searched comprehensively for systematic reviews?
- Specify search terms clearly
- Conduct search in more than one database
- Search the ‘grey’ literature (abstracts, conference proceedings, unpublished theses, etc) to avoid publication bias
- Studies that detect a significant difference are more likely to be published.
How are relevant articles selected for systematic reviews?
- Many of the retrieved articles will not be relevant:
- Studies using populations or study design differing from eligibility criteria (i.e., animal studies, non RCT)
- Studies that include an additional intervention (i.e., vitamin D and calcium together)
- Studies that do not include a measure of your outcome of interest
- Document reasons for exclusion!
How is a data set assembled comprehensively for systematic reviews?
- Tabulate data from included studies:
- study population characteristics
- sample size
- specific intervention
- duration of follow-up
- outcomes measured
- etc…
How is the quality of included studies assessed for systematic reviews?
- Assessment criteria should be specified
- E.g., randomization, blinding
- Can use a checklist
- ADA checklist to assess research papers
- CONSORT
- At least 2 reviewers should do this independently
How is the quality of included studies assessed for systematic reviews?
- Assessment criteria should be specified
- E.g., randomization, blinding
- Can use a checklist
- ADA checklist to assess research papers
- CONSORT
- At least 2 reviewers should do this independently
Why is meta-analysis conducted?
Combines the results of individual studies to increase the overall sample size → improves the statistical power of the analysis
What are treatment effects considered for meta-analyses?
Determination of the treatment effects for each study.
- Odds ratio (OR)
- Relative risk (RR)
- Mean difference → (mean group 1 - mean group 2)
- 95% confidence interval
When is OR, RR, or mean difference significant?
OR, RR → significant if CI does not cross/include 1
Mean difference → significant if CI does not cross/include 0
How are meta-analyses conducted?
- Determine treatment effect for each study (OR, RR, mean difference, 95% CI)
- Calculate overall treatment effect as a weighted average of individual treatment effects (weighting gives more emphasis to larger individual studies)
- Display data.
How are forest plots interpreted?
- Each ‘black box’ shows the OR (RR or mean diff) for an individual study → the lines extending from the box shows the 95% CI
- The size of the black box reflects the weight of the study in the meta-analysis (and usually, the sample size)
- The clear weighted diamond is the weighted mean of all the studies
Of all the 9 studies in this forest plot, how many showed a statistically significant effect of the treatment?
3
Why is heterogeneity assessed?
Are results from individual studies, in general, consistent?
- Study results may differ due to differences in:
- Characteristics of the participants
- Study design (length of intervention, dose)
- Outcome measurements
- If substantial heterogeneity exists it may not be appropriate to combine studies.
- Analysis may be repeated in subgroups to see if heterogeneity is improved.
How can we test for heterogeneity?
- Look at forest plot, CI should overlap
-
Chi2 (or Cochrane’s Q)
- Are differences between studies due to chance alone?
- If significant (P<0.05), there is heterogeneity between studies
- Are differences between studies due to chance alone?
-
I2
- Percent variation in studies due to heterogeneity
- Generally >50% = substantial heterogeneity
What happens if there is heterogeneity between individual studies in a systematic review? [4]
Don’t do meta-analysis
Remove studies
Sensitivity analysis with subsets of studies
Account for it in a ‘random-effects’ model
How is publication bias assessed?
Funnel plot
How is a Funnel Plot interpreted?
- The vertical axis is a measure of the study size or precision of the study (e.g., standard error)
- The larger and more precise (i.e., less variable) the study, the further up it is placed on the plot
- The horizontal axis measures treatment effect
- An approximately symmetrical distribution of studies on either side of the overall effect suggests low bias
- Un-symmetrical distribution of studies suggests a publication bias.