L7 - Meta Analysis

Question 1

What are the 4 types of review types?

Answer 1

Literature Review

Scoping Review

Systematic Review

Meta-analysis

Question 2

What are the features of a literature review?

Answer 2

Can be “narrative” or “qualitative”

Not expected to be systematic or overly descriptive in methods

Often Published by ‘experts’ in the field

Question 3

What are some of the flaws in literature reviews?

Answer 3

Researchers may put emphasis on studies that other researchers see as ‘less important’. Meaning different researchers can reach different conclusions based on the literature.

Question 4

What are the key features of a scoping review?

Answer 4

Covers key concepts

Sources and types of evidence are available

Explores complex areas or areas not previously reviewed comprehensively or rigorously

Question 5

What review would you use for the following research questions?

1. In what way has the concept of resilience been used in the literature concerning the transition to parenthood?
2. To what extent are researchers using theory to underpin their work?

Answer 5

Scoping Review

Question 6

What are the key features of a systematic review?

Answer 6

1. Review that has been conducted in a systematic manner
2. Is set-out like other studies: intro, methods, results, discussion, qualatative study of the results of empirical / original research
3. Differs from a narrative review: it is comprehensive/exhaustive, rates methodological quality of studies, can include effect sizes.
   * Important in evidence based practice, very rigorous and documented methodology. Aim is for the research to be reproduced and updated by other researchers.*
   * e.g. looking at the effectiveness of a intervention.*

Question 7

What review would we use for the following research questions?

How has motivation interviewing been used for adults with multiple sclerosis?

What is the potential impact of delivery format an/or modality on intervention effects?

Answer 7

Systematic Review

Question 8

What are the key features of a meta analysis

Answer 8

1. Statistical analysis of data obtained from empirical studies
2. Quantitative data from multiplke studies are combined and analysed
3. Main statistic is effect size
4. There are different effect sizes - it all depends on the study design/data you need to extract

Question 9

What is the main goal of a meta-analysis?

Answer 9

To discover how big an effect size actually is,

and what factors might moderate and influence that effect.

Question 10

What type of review would we use for the following research questions.

How do mean burnout levels reported by postgraduate trainees compare to norms?

Are there differences in burnout patterns, based on training level (interns, residents, registrars), and speciality (e.g. surgical vs. non-surgical)

Answer 10

Meta-analysis

Question 11

What are the 10 steps of a meta-analysis?

Answer 11

1. State Research Question
2. Do a literature search
3. Decide on objective criteria
4. Screen studies
5. Extract information
6. Assess study quality
7. Calculate the effect sizes
8. Do the meta-analysis
9. Write it up
10. Publish

Question 12

What is the reporting standards set by the APA for meta-analysis called?

Answer 12

MARS

“Meta-analysis Reporting Standards”

Question 13

The first step to undertaking a meta analysis is to state the research question

What is the criteria we use for stating the research question?

Answer 13

PICO

Population

Intervention (or exposure)

Comparator

Outcome

e.g.

P: adults, both sexes, all ethnicity

I: Mindfulness intervention

C: Psychotherapy

O: Anxiety symptoms or generalized anxiety

Example question: For adults, does mindfulness meditation compared to other psychotherapies reduce anxiety?

Question 14

What is the most important thing to do when using the PICO criteria when stating the research question

Answer 14

Each step of PICO needs to be defined

If possible, justify the definitions

Question 15

Step 2 in a meta-analysis is “do a literature search”

How should you go about doing this?

Answer 15

Being a thorough as you can be.

• Search electronic databases that have addressed the same research questions*
• Search conference proceedings, dissertations, Google Scholar etc.*
• Consult experts in the field*
• Search the reference lists of articles you have found*
• Search the reference lists of relevent reviews*

Question 16

When you are doing your literature review, you will find from 1000 - 10,000 studies.

How do you reduce this into something manageable?

Answer 16

Decide on an objective criteria (step 3 in generating a meta-analysis)

Decide on an inclusion criteria for studies, and you need to justify these criteria, be objective and be explicit

Question 17

Step 3 in doing a meta analysis is “decide on objective criteria”

What is important to do in deciding on this criteria?

Answer 17

The criteria needs to be justified, objective and explicit.

• example criteria:*
• how have you defined key variables?*
• Which participants are eligible (so reader can assess to whom the results apply)*
• Are specific study designs eligible?*
• Is ther ea time period in which studies needed to be conducted?*

Question 18

How do you make the process of finding studies easier in a meta-analysis?

Answer 18

Screen Studies (Step 4)

Question 19

Step 4 of conducting a meta-analysis is to “screen studies”

What steps can you use to do this?

Answer 19

Consider screening tools

How were the studies screened? (i.e. title, abstact, full text)

Number of reviewers involved in search process (if > 1 reviewer, how were disagreements resolved)

Number of studies included

Number of studies excluded (and reasons why)

Question 20

Step 5 of conducting a meta-analysis is “extract information”

What are some examples of the kinds of information should you be extracting in this phase?

Answer 20

Author name
 Year of publication
 The population
 The study design (RCT? Observational? etc)
 The intervention and comparison conditions (if relevant)
 The outcomes and how were they measured
 No. of individuals (intervention vs. control? Total N?)
 Effect size data (e.g. means, SDs, N, r….depends on study design)

Question 21

Step 6 when conducting a meta-analysis is “assess study quality”

What does this mean?

Answer 21

Critically appraise the information contained within your meta-analysis

Question 22

What is a good way to assess data (step 6) in your meta-analysis?

Answer 22

Using tables

Question 23

What is this an example of?

Answer 23

Overall quality assessment of the studies involved in a meta-analysis

Question 24

Once you have collected all your studies in your meta-analysis, what is the next step for you to take? (step 7)

Answer 24

Calculate the effect sizes

Question 25

The effect size you will use in your meta-analysis is dependent on...

Answer 25

Your research question

Question 26

What effect size should you use when examining an association?

Answer 26

Correlation *r*

Question 27

What effect sizes measures should you use when comparing two groups with **dichotomous outcomes?**

Answer 27

**risk ratio** or **odds ratio**

Question 28

What effect sizes measures should you use when comparing two groups with **continuous outcomes?**

Answer 28

**Cohen's *d*** or **Hedges' *g***

Question 29

Step 8 in a meta-analysis is **do the meta-analysis** Before you can combine different effect sizes from different studies, you need to look at a number of particular issues. What are they? (4 in total)

Answer 29

1. How were studies with more than one effect size handled? 2. Were fixed or random effects models used, and why? *(how do you weight the effect sizes across studies)* 3. How was heterogeneity assessed? *(how different were effect sizes in terms of magnitude)* 4. How was publication bias assessed?

Question 30

One important aspect of conducting a meta analysis is to ensure **data independence** What does this mean?

Answer 30

If one study contibuted **more than one effect size** this means that the same participants provided multiple outcomes for your review. *The measures for the study are not **independent,** and it would be inappropriate to treat each effect size as independent*

Question 31

What might you do if a study has more than one effect size (e.g. not data independent)

Answer 31

Average the effect sizes for that study.

Question 32

Some meta-analysis **weight** the effect sizes, what does this mean?

Answer 32

They put more weight on the effect size found, depending on the sample size and variance of the study. ## Footnote *Studies with larger study size are given a bigger weight in the overall effect estimate, as they are considered more precise.*

Question 33

When pooling or combining the effect sizes from different studies for a meta-analysis, there are two options you can choose from. What are they? ***important for exam***

Answer 33

**Fixed** or **Random** model **Fixed effect model:** assumes that effect sizes are similar across studies *(homogenous in terms of outcomes, population examined etc.)* **Random effect model**: asssumes that effect sizes vary from study to study *(effect sizes might be higher or lower when participants are older or more educated than other studies)*

Question 34

What do the following 4 studies assess? Q statistic Tau (equivalent to SD of an effect size) I² (expressed as a percentage) Subgroup analyses (note: recommend at least 4 studies per subgroup) and/or meta-regression.

Answer 34

Estimate variablility of effect sizes across studies **i.e Heterogeneity**

Question 35

When assessing heterogeneity of effect sizes in your meta-analysis, what might happen if you have a low number of studies (e.g. 10 or less)

Answer 35

It will have **low power**

Question 36

One important issue in a meta-analysis is the assess **publication bias**, what is it? ## Footnote *important for exam*

Answer 36

Refers to the fact that significant findings are more likely to be published in journals than nonsignificant findings

Question 37

The file-drawer problem is another name for...

Answer 37

publication bias

Question 38

What are two methods you can use to test **publication bias?**

Answer 38

**Fail-safe N (N_fs)** **Funnel plot analysis**

Question 39

What does the **fail-safe N (N_fs)** tell us?

Answer 39

The statistic represents the number of hypothetical studies that woul dbe required to reduce a given *r* to a small, non-significant effect (i.e. *r* = .10)

Question 40

What is a **funnel plot** analysis? How does it work?

Answer 40

A scatter plot of the effect estimates of individual studies. These effect estimates are plotted against some measure of each study (e.g. size, or precision, or the standard error of effect estimate) *in the example, each dot represents an effect size from a single study. Studies with larger sample sizes are towards the top and smaller studies at the bottom. In the absence of publication bias, the dots should be relatively symmetrical. When there is publication bias, there are gaps in the funnel (such as the right image)*.

Question 41

What is the issue with using a funnel plot for looking at publication bias?

Answer 41

It is visual (very subjective) * However, there are inbuilt tests in meta-analysis that confirm the results of the funnel plot.* * If there are not enough studies, the power of the study is small.*

Question 42

Step 9 of doing a meta analysis is "**writing it up**". What are the features within writing it up?

Answer 42

Tables and/or forest plots to summarise: *o characteristics of the studies (eg. sample ns, country, study design etc) o effect size estimates, including p values and CIs*  Statement of major findings  Compare/contrast to available research  Critically evalute findings - Alternative explanation for observed results  Generalisability of results (was there bias present)  Limitations of review  Implications for future research, theory, policy, practice