L7: Principles Of Critical Appraisal Of Research

Question 0

The scientific method

Answer 0

• the order of steps is important
• ideally the experiment has only one variable, therefore any effects can be attributed to that variable
• conclusions based on quantifiable and reproducible evidence
• control of variables other than that of interest -> confidence in attribution of effect

Question 1

Why study research design?

Answer 1

• minimize potential for bias or unclear interpretation

- maximize efficiency of resource utilization

Question 2

Sampling principles

Answer 2

• random sampling: all population has an equal probability of selection
• samples may be technically non-random but arbitrary
• in practice, samples are often obtained for convenience, higher chance of being non-representative
• applies to surveys as much as experimental studies

Question 3

Measurement issues

Answer 3

• most closely match RQ/hypotheses
• best (psychometric) properties: reliability/validity, min variance, most responsive (to experimental factor
• most feasible, least cost

Question 4

What is critical appraisal?

Answer 4

• balanced assessment of benefits and strengths of research against its flaws and weaknesses
• assessment of research process and results
• consideration of quantitative and qualitative aspects of research
• to be undertaken by profs as part of their work
• is inherently retrospective

Question 5

Key assessment points

Answer 5

• clarity of RQ: quality of sampling (appro to pop), quality of measurement (appro to constructs)
• quality of experimental design (min bias, max power)
• sample size (power)
• appro of statistical analyses and interpretation

Question 6

Critical appraisal skills program (CASP)

Answer 6

1. Did the study ask a clearly defined question?
2. It is a rct?
3. Subjects appro allocated groups?
4. Staff and subjects blind?
5. All subjects who entered accounted for?
6. All subjects followed-up consistently?
7. Sample size adequate for power?
8. How are the results presented and key message?
9. How precise are the results?
10. Were all important outcomes considered?

Question 7

Biases in appraisal guidelines

Answer 7

• primarily geared to intervention studies, CBT, pharmacotherapy
• less attention given to other research
• seeking scientific evidence of potential effects attributable to intervention
• rcts considered the holy grail

Question 8

Retrospective power analysis

Answer 8

• B/w group power curve
  • prospective power for comparing groups gen less impressive
• within-group comparisons gen have more power

Question 9

Journal quality

Answer 9

• rated by prestige or reviewer difficulty

rated by impact factor

• notionally a measure of the impact articles published in a given journal have in their field on average
• # cites of articles/ # articles
• measure quality? : consider discipline with small numbers of researchers, cannot complete with psych
• can the IF mislead? Some types of articles are more likely to he cited than others -> higher IF

Question 10

Quality of scores

Answer 10

• take appraisal criteria and assign a score to each. Sum scores for an article to obtain overall score

Pro

• quantifies quality
• enables comparison across studies
• regardless of journal IF or quality

Cons

• assumes we all apply same weight to a given item
• a score could fail certain items but still get an overall pass

Largely discarded today