Lecture 2: Effect sizes and a-priori power analyses

Question 1

What is the main limitation of statistical significance?

Answer 1

When there are many participants, the results are easily significant which does not directly address how large or clinically significant an effect is as the criterium for significance is arbitrary.

Question 2

What are effect sizes?

Answer 2

standardized measures of how large an effect is. These are the types for continuous outcome measures:
* Pearson’s r (.1 = small, .3 = medium, .5 = large)
* Cohen’s d (0.2 = small, 0.5 = medium, 0.8 = large)
* Hedges’ g (0.2 = small, 0.5 = medium, 0.8 = large)

Question 3

What is Cohen’s d (check slide 20)?

Answer 3

(y1-y2)/ sdpooled

Question 4

What is Hedges’ g (check slide 20)?

Answer 4

Cohen’s d * J

Question 5

What is the difference between standard deviation?

Answer 5

Sd= √var
Se= √var/N

Question 6

Which d’s are common in intervention research?

Answer 6

d= 0.8 for intervention vs waiting list control
d= 0.5 for intervention vs other intervention
-> difficult to grasp how clinically meaningful an effect is

Question 7

How to overcome issue for clinical meaning?

Answer 7

By standardizing scores to a population norm and establish a cut-off point of being recovered. Also by using effect sizes for discrete outcome measures:
* Risk ratio
* Odds ratio (1.5 = small, 3.5 = medium, 9 = large)
* Number Needed to Treat (NNT)

Question 8

Odds ratio

Answer 8

Divide recovered by not recovered for each condition. This means that you are x times more likely to recover than not to recover after one condition than the other condition

Question 9

What are the limitations for discrete measures?

Answer 9

• effect sizes are warped
• comparability across studies is limited
• standardized scores can become more abstract

Question 10

What is the role of effect sizes?

Answer 10

• Provide a standardized measure of the strength of an effect
• Allow to draw conclusions across multiple studies (meta-analysis)
• Help to calculate how many subjects you need when planning
  a new study (power analysis)

Question 11

What is an a-priori power analysis used for?

Answer 11

To determine how many subjects are needed to get with a reasonable chance (80%) a significant result

Question 12

What is needed to determine the required N?

Answer 12

• significance level/alpha (usually .05)
• effect size that you expect (e.g. d = 0.8)
• desired power (usually 0.8)

Question 13

Why is the selection criteria of clients a shortcoming?

Answer 13

There is low comorbidity and less complex forms of psychopathology. There could be low within group variance (could result in an higher F value when not correct). Results thus mat not generalize-> generalization crisis in psychology. Solutions: meta-analysis and N=1 study

Question 14

How is RCTs focusing on specific symptom outcome measures an issue?

Answer 14

Does not capture the key problems or cause of clients-> broader issue of measuring latent constructs in psychology. Solutions: measures like quality of life and other clinical significance measures

Question 15

How is RCTS reporting effects on various outcome measures an issue?

Answer 15

Multiple tests can increase the probability of significant results and so there are different effects in a study (some significant, some not)-> analytic flexibility issue in psychology. Solutions: register RCT with a priori hypotheses and meta-analysis

Question 16

Why is it an issue that RCTs focus on pre-post difference in symptoms?

Answer 16

It does not tell you the mechanism why the intervention works and does not account for non-linear change processes. Solutions: study mediators, N=1 study

Question 17

What are shortcomings of clinical practice?

Answer 17

1. Overconfidence in clinical impression and tailored interventions as reliability is low and unclear if/how treatment works. Big issue in complex decision-making-> look at moderators
2. No systematic evaluation as difficult to tell when to stop a treatment, and knowledge stops when therapist retires-> evaluate intervention systematically, N=1 study

Question 18

Why are both RCT’s and clinical info needed?

Answer 18

RCTS provide empirical basis on effectiveness of therapy and clinical experience is needed to monitor treatment in real world

Question 19

What is the relationship between statistical significance and sample size?

Answer 19

Statistical significance is strongly determined by the number of subjects. As the sample sizes increases, the result is more likely to become statistically significant.
Because of this, even a small effect can be statistically significant with a large sample size.

Question 20

How would using the pooled standard deviation affect the effect size?

Answer 20

The standard deviation of the control group is smaller (SD = √3) than the pooled standard deviation of both groups (SD = 2). By using the smaller SD of the control
group, Cohen’s d will become larger.

Question 21

How to interpret AUC?

Answer 21

This means that when a subject from the intervention group is compared to a subject from
the control group, in x% of the cases, the subject from the intervention group is classified
as having a better outcome than the subject from the control group.

Question 22

Why is it needed to perform an a-priori power analysis?

Answer 22

A limitation of too few
subjects: high probability that the result is not significant, while there may be a true effect (Type II error). Limitation of too many subjects: many patients unnecessarily receive a treatment that may not be effective or may have side effects (= unethical); wasted research
recourses (i.e., research hours, money); trivial differences might become significant.

Question 23

How does a smaller alpha value affect sample size?

Answer 23

The critical F-value also becomes larger, and to exceed this threshold a larger sample size is needed to get a higher F-value.