Assessing Research Flashcards
Why is critical appraisal important?
Our ability to discern correct information or misinformation is limited by our ability to understand if something is reliable.
What does the reproducibility crisis show?
Just because something is published in a paper, doesn’t mean its accurate / valid.
For things to be real / true, you should be able to find it multiple times, contexts, groups of people, etc.
What is the scientific method, and what is it vulnerable to?
Formal process we go through to conduct science - important for the conclusions we draw from the studies we have done.
Internal validity threats: are we isolating the effect of the IV on the DV, and are we measuring what we intend to measure? (i.e. if there’s no control group?)
External validity threats: how do the findings generalise? What is the broader applicability / generalisability of the findings?
What is the most common reason that papers are retracted over time?
In 2014, 946 papers were retracted, 411 for fraud.
The most common reason is fraud.
Outline critical appraisal:
- balanced assessment of strengths against limitations
- process of research and results of research
- clarity of research question, quality of design, appropriateness of statistical analysis, appropriateness of interpretation
Do we ‘believe’ an effect? Do we build our future research on these findings? DO we endorse a particular therapy / intervention for your client?
What are some guidelines for the Assessment of Quality?
CASP: Critical Appraisal Skills Guidelines (assessing specific types of studies)
- A) are the results valid? (clear focussed issue, random assignment, drop out rates, blind to treatment, baseline group comparability
/ similarity)
- B) What are the results?
- C) Will the results help locally?
and
Cochrane’s Risk of Bias Tool (Specific to RCTs) - looks at source of bias, support for judgement
What are some article reporting guidelines?
CONSORT: consolidated standards of reporting trials
APA JARS: Journal article reporting standards
Guidelines for the assessment of strength of evidence for specific interventions:
NHMRC evidence hierarchy: National Health Medical Research Counsel
- The highest point of evidence is a systematic review of randomised control trials
- The second highest are randomised control trials
How will your perspective affect your appraisal?
Perspective will affect which aspects of appraisal you give most weight to?
i.e. are you reading the article for research direction, to input into clinical practice, for personal interest?
Quality assessments - what not to do:
Do not take appraisal criteria and assign a score to each, and then sum scores to obtain overall score.
Pros: quantifies quality, enables comparisons across studies
But the cons outweight:
- assumes we all apply same weight to a given item
- a score could fail certain key items but still get an overall pass
Taking into consideration journal quality:
- peer review should reduce bias in published research
- journals each have an IMPACT FACTOR (measuring how much each of the papers in the journal are cited by other studies - the inCites Journal citation reports)
The highest rank in annual review of psych had an impact factor of 23.6- the annual review of psychology!
What does IF (impact factor) measure?
Does IF measure quality?
- some disciplines are going to have more research be done and more opportunities for citation
- some papers may be cited as critique (i.e. not supporting)
- not all citations are good citations
- some types of IF exclude self-citations
Can the IF mislead?
- some kinds of articles tend to be cited more than others (such as review articles)
- journals can act to increase their IF
