January 20th Flashcards
CRITICAL EVALUATION OF PUBLISHED RESEARCH-
despite peer review, validity of design or conclusions not guaranteed
ultimately health professional responsible for judging validity and relevance of published material
proper attitude is resolute skepticism, due to:
probabilistic and provisional nature of science
investigation of complex phenomena
ethical or economic constraints to desired levels of control
aim of critical evaluation:
identify strengths and weaknesses of research publication
ensure patients receive assessment and treatment based on best available evidence
Examine:
criteria for critical evaluation of quantitative research paper
implications of identifying problems in design, measurement and analysis in a publication
strategies for summarizing and analyzing evidence from a set of papers
implications of critical evaluation of research for health care practice
CRITICAL EVALUATION OF PUBLISHED RESEARCH
despite peer review, validity of design or conclusions not guaranteed
ultimately health professional responsible for judging validity and relevance of published material
proper attitude is resolute skepticism, due to:
probabilistic and provisional nature of science
investigation of complex phenomena
ethical or economic constraints to desired levels of control
CRITICAL EVALUATION OF THE INTRODUCTION
inadequacies may signal research erroneously conceived or poorly planned
Adequacy of the literature review
sufficiently complete to reflect current state of knowledge in the area
no significant omissions relevant to the presented research
unbiased in presenting unfavorable points of view
Clearly defined aims or hypotheses
clearly and operationally stated
if lacking, conceptual advances ambiguous
Selection of appropriate research strategy
strategy appropriate to aims
eg. Survery inappropriate for demonstrating causal effects correlation, not causeexperimental instead
Selection of appropriate variables
operational definition of variable should be suitable to phenomenon studied
Internal Validity
ability to attribute differences or changes observed to the independent variable
External Validity –
extent to which results can be generalized to other samples or situations
Subjects
– shows if sample representative
Sampling model – unbiased, optimizes representation?
Sample size – appropriate to heterogeneity of population?
ensures representative
important for statistical power
Description of the sample – clear description of characteristics/variables (eg. demographics)?
Validity and reliability
used standardized apparatus or measurement tool?
established reliability and validity of new apparatus?—compare to another instrument or method
Description of measurement tool
full description of novel instrumentation?
allows to be replicated
methods- procedure
full description necessary for replication and evaluation of internal/external validity
Adequacy of the design
controls for extraneous influences?
ie. minimizes threats to internal validity
should control for alternative explanations of the data
Control groups
used (eg. placebo, no treatment, conventional treatment)?
controls for extraneous effects
if not used, internal validity questionable, size of effect difficult to estimate
mwethods Procedure (cont.) Subject assignment
avoids initial differences between subject groups?
also important in quasi-experimental or natural comparison studies
Treatment parameters
describes all treatments to different groups?
treatments have same intensity and applied equally by different personnel?
Rosenthal (bias) and Hawthorne (expectancy) effects
ie. if experimenters/observers or subjects aware of aims and predicted outcomes
uses single or double blind design?
methods- procedure
Settings
describes sufficiently to evaluate generalizability?
has implications for external validity (ecological)
Procedure (cont.)
Times of treatments and observations
clearly indicates sequence of treatments and observations?
variability in treatment and observation times can affect internal validity
controls for or considers series effects? (eg. that pre-tests produce effect)
CRITICAL EVALUATION OF THE RESULTS
statistically correct summary and analysis of the data?
inadequacies could have produced erroneous inferences
complete summaries of all relevant findings?
Tables and graphs
correctly tabulated or drawn?
adequately labeled for interpretation?
results: Selection of statistics
appropriate descriptive and inferential statistics used?
if not appropriate could distort findings/inferences
Calculation of statistics
no errors?
computers generally ensure
CRITICAL EVALUATION OF THE DISCUSSION
draws inferences from data in relation to aims or hypotheses
if inferences incorrectly made, conclusions may lead to useless or dangerous treatments being offered
Drawing correct inferences from the data
considers limitations of descriptive and inferential statistics?
Eg. correlations don’t necessarily imply causation
Eg. lack of significance could be type II error (miss of effect)
Logically correct interpretations of the findings
interpretations follow from statistical inferences, without introducing extraneous evidence?
Eg. n = 1 design – shouldn’t claim procedure generally useful
Protocol deviations
indicates and takes into account unexpected deviations from intended design?
Eg. placebo/treatment code broken
Eg. “contamination” between control and treatment groups discovered
researchers obligated to report so that implications for results can be considered
Generalization from the findings
data from sample only generalized to that population?
frequent tendency to generalize to subjects or situations not considered in original sampling
Statistical and clinical significance
appropriate conclusion of clinical applicability/significance?
should consider size of effect, side-effects, cost effectiveness, value of outcome
Theoretical significance
relates results to previous relevant findings?
unless logically related to the literature, theoretical significance of findings unclear
OVERALL EVALUATION
even if investigation flawed, may be useful information to be drawn
negative results are also useful
Journal of Negative Results in Biomedicine (est. 2002)
Journal of Negative Results in Ecology and Evolutionary Biology (est. 2004)
Text – Table 23.1 Checklist
CRITICAL EVALUATION OF THE LITERATURE: META-ANALYSIS
unusual that results of a single research publication sufficient for clinical decision
need to consider multiplicity of papers
Review – critical summary of literature and implications
Meta-analysis – systemic procedures to summarize overall implications of set of papers
To review/evaluate literature:
- identify relevant literature (Section 2 – Research Planning)
- evaluate critically the key papers (as discussed in this Section)
might reject some if have irrepairable errors or don’t fit selection criteria - identify patterns of findings in the literature
tabulate (table listing the previous findings in the literature) - identify crucial disagreements and controversies
- propose valid explanations for disagreements
provide theoretical framework for resolving controversies, proposing future research
Two main strategies for summarizing findings from multiple papers:
a. Quantitative – condense results from several papers into a single statistic
represents average effects size
b. Qualitative – tabulate key features of related publications
Eg. designs, subject characteristics, measures used
allows relating differences in features to outcomes
Qualitative comparison (cont.)
tabulates key info and outcomes
enables emergence/demonstration of pattern
clear pattern doesn’t always emerge
conflict might emerge about nature and causes of findings
eg. Table 22.2 – large difference between results of Smith and Jones versus those of Brown and Miller
explanation not necessarily true, but hypothesis to guide future investigations
