Doing Psychology 7.2

Question 1

What is meta-analysis? (definition + 2 points)

Answer 1

Statistical method of combining the results of at least 2 primary studies carried out for the same general purpose
- not a summary of related literature
- a type of research that tests hypotheses and leads to valid conclusions, to revision and proposal of theory

Question 2

Why conduct meta-analysis? (3 points)

Answer 2

• Combining primary studies = result based on large sample size = more precise estimate of effect
• Considering all studies = bigger picture = avoids reliance on one primary study
• More objective / transparent than a narrative review, follows set of standardised procedures that are documented in a research report, less risk of researcher bias

Question 3

What are three conditions for meta-analysis?

Answer 3

Primary studies must be quantitative
Studies must address same constructs and effect
Must be at least two studies

Question 4

What are the 5 steps involved in a meta-analysis?

Answer 4

1. specify RQ, search criteria, study eligibility
2. literature search, retrieve studies, filter down through eligibility criteria
3. extract results from eligible studies and code study features
4. compute and combine effect sizes into single summary
5. explore heterogeneity of effect sizes, moderators of effect sizes, publication bias

Question 5

How is effect size estimated, and why? (4 points)

Answer 5

• To combine results from studies, results need to be converted to same metric
• Effect sizes express quantitative results in terms of standardised metrics that can be compared across studies
• Many different effect sizes available but only handful used in practice
• Best choice depends on design of studies

Question 6

What are three commonly used effect sizes?

Answer 6

Standardised mean difference (Cohen’s D)
- comparing groups on a continuous outcome variable
- mean difference as a proportion of standard deviation

Correlation coefficient (Pearson’s r)
- for associations between two continuous variables

Odds ratio, hazard ratio, relative risk
- for binary variables

Question 7

How are the studies combined? (3 points)

Answer 7

Common metric of effect size is extracted from each study
Each study receives a weight
An overall effect is calculated

Question 8

What is the formula for the weight of single study effect size and what does this mean for larger sample sizes?

Answer 8

Weight = 1 / Standard Error ^2
Standard error decreases as sample size and precision increases
Larger studies receive larger weights

Question 9

How are effect sizes most commonly displayed?

Answer 9

Forest plots

Question 10

What is heterogeneity of effect sizes and what can it be caused by?

Answer 10

Effect sizes that are different across studies
- different populations studied
- different study designs
- different treatment variations, duration, dose, level of experience of practitioners
- different measures e.g. scales used, duration of follow-up
- differences in quality of study (systematic biases)

Question 11

What are two types of sub-groups?

Answer 11

Participants e.g. males/females
Studies e.g. geographical location

Question 12

What is publication bias, why does it occur, and how does it ply into meta-analysis? (4 points)

Answer 12

• Not all studies translate into a publication: peer review may select out studies with non-significant results, authors of studies with non-significant results may be less motivated to publish
• If studies reporting some results are not getting published, full story is not seen
• Overall effect size measure will overestimate size of effect
• Can examine whether publication bias is present among studies in meta-analyis

Question 13

What are the 2 main issues with finding sound studies for meta-analysis and how can they be prevented? (Garbage in-garbage out)

Answer 13

• Methodologically sound studies not always used
• Quality of reporting in primary studies not always holistic
  –> seek additional info by emailing authors + exclude poor quality studies + group studies by quality and compare results

Question 14

How can studies in meta-analysis become comparable? (Comparing apples and oranges) (2 points + solution)

Answer 14

• Need to be sufficiently similar (answer the same question + comparable methodologies)
  BUT this is subjective
• Including studies that are too different might obscure real effects
  –> SOLUTION: assess heterogeneity + study its sources

Question 15

How can meta-analyses account for studies that do not get published? (File-drawer problem) (2 points)

Answer 15

Seek out unpublished studies
Estimate magnitude of problem e.g. funnel plots

Question 16

What are 4 main limitations of meta-analysis?

Answer 16

• More objective than narrative review but still many subjective decisions e.g eligibility criteria
• Might miss / need to exclude some studies
  e.g. if effect sizes can’t be calculated from info in paper
• Some designs = difficult to compute effect sizes from (impossible for qualitative studies)
• Focuses on ‘bigger picture’ BUT may miss important nuanec

Question 17

What is a summary of what meta-analysis is, its benefits, and its limitations?

Answer 17

• Provides quantitative summary of effects of multiple single studies