Effect sizes and power

Question 1

What is the difference between effect sizes and significance?

Answer 1

Significance shows the presence or absence of an effect, but effect size gives an indication of the magnitude of this effect.

Question 2

What are the advantages of ES over Significance?

Answer 2

1. ES is not affected by sample size, whereas significance does.
2. ES helps us to compare significant effects.
3. ES gives meaning to nonsignificant effects.

Question 3

Which measures of ES do we use when comparing 2 conditions? And which aspects are considered in this computation?

Answer 3

Cohen’s d (measuring the population ES), Hedge’s g (sample ES) and r (equivalent of correlation coefficient; also named R2). We always mention Cohen’s d, while we actually always calculate Hedge’s g (= Ma - Mb / SDsample).

Question 4

Which measures of ES are used to describe explained variance?

Answer 4

1. Eta squared n (η2) = SSeffect/SStotal
2. Partial eta squared = SSeffect/(SSeffect+error) > takes out the other factors, so gives a larger effect than η2.
3. R-squared = SSregression/SStotal

Question 5

For which statistical tests are (partial) eta squared and r-squared mostly used?

Answer 5

(Partial) η2 is mostly used for ANOVA, R-squared is mostly used for regression.

Question 6

Which ES-measures can be used in non-parametric data?

Answer 6

1. Ranks for group comparison (ordinal scaled data): rank = z/√N.
2. Phi (φ) for categorical data (chi-square tests): φ = √(X2 /N)

Question 7

Why do we use √N in the computation of ES of nonparametric tests?

Answer 7

By dividing through √N, you correct for the sample size, since you don’t fulfill the assumptions as in parametric tests and thereby the ES of nonparametric tests are influenced by sample size.

Question 8

Why do we use CI’s for effect sizes?

Answer 8

CI’s show the reliability and significance of ES (e.g., if the interval contains the critical ES)

Question 9

What is the probability of superiority (PS)?

Answer 9

This is the % of occasions when a randomly sampled member of the distribution with the higher mean (clinical group) has a higher score than a randomly sampled member of the other distribution (healthy controls).

Question 10

What is the percentage of nonoverlap between 2 distributions (u)?

Answer 10

Ja spreekt wel voor zich he, laat zien hoe goed een test kan discrimineren tussen twee groepen

Question 11

Why should you use the PS and U in describing ES?

Answer 11

These concepts help the reader understand the meaning of an effect.

Question 12

What is the relationship between N, ES and power?

Answer 12

Small effects have low power and require large sample sizes; large effect sizes have higher power and will therefore need less large sample sizes to demonstrate the effect.

Question 13

Why is it important to control for confounding factors?

Answer 13

To prevent oversimplification of an effect; you can’t just attribute a difference between groups to the presence/absence of pathology.

Question 14

How can you control for confounding factors?

Answer 14

1. Use a control task (e.g., task that requires different funcitons, but same setting/task difficulty) to demonstrate a specific impairment.
2. Matching in the sample to control for possible confounders
3. Include another control group.
4. Being careful with attributing differences between two groups to the mere presence or absence of pathology.

Question 15

When do you use ANCOVA?

Answer 15

When you can’t find a suitable control group and you still want to control for confounders.

Question 16

What is the Lords’ Paradox?

Answer 16

You can’t control for variables that are fixed (such as gender, age, length). When you still do this, you will get biased results > so you need to apply a quasi-experimential design such as ANCOVA

Question 17

What are the assumptions of ANCOVA?

Answer 17

1. Independence of covariates and IV’s

2. Homogeneity of regression slopes: the b-coefficient of the covariates must be equal across all subpopulations

Question 18

Why are there different categorizations for ES (Cohen’s, Rogers)?

Answer 18

Because the magnitude of the ES depends on the context; for diagnostic purposes, you need far more strong effects than for treatment purposes.

Question 19

Why and when should you use pilot studies?

Answer 19

You should only use pilot studies to test the feasability of the study design (NOT TO ESTIMATE THE ES); then you can still improve the design before performing the actual study.

Question 20

Which mistake is often made with post-hoc hypotheses and how should this be solved?

Answer 20

Post-hoc hypotheses are often tested on the same data as a priori hypotheses, leading to errors is the ES and p-values; you should only design post-hoc hypotheses based on an exploratory analysis for generating knowledge for future studies!!

Question 21

What is the goal of exploratory studies? and what are the 3 main characteristics?

Answer 21

Generating knowledge to foster the generation of strong hypotheses in future studies.

1. Report NO p-values
2. NO hypotheses testing
3. NO a priori hypotheses

Question 22

What is the difference between Cohen’s categorization of ES in comparison to Rogers’?

Answer 22

Cohen’s ES categorization is used for identifying differences between two groups while Roger’s ES categorization is used to identify individuals that feign symptoms relative to genuine patients.

Question 23

What is the “file-drawer problem”?

Answer 23

Many results remain unpublished, and then negative/nonsignificant results in particular; this leads to publication bias.

Question 24

What is the relationship between the lack of exploratory studies and poorly designed hypotheses-testing studies?

Answer 24

Exploratory studies are needed to generate strong hypotheses; since these studies are lacking, hypothesis-testing studies are often weak because they don’t have the info that is needed for strong hypotheses, leading to poor study designs.

Question 25

What is the problem with post-hoc power calculations?

Answer 25

Post-hoc power calculations are often used by researchers to “prove” that their nonsignificant results are due to low power instead of asking the wrong questions(hypotheses) in the first place.

Question 26

Make the right sequence of events: test hypotheses in a real study - designing hypotheses - conduct pilot study for feasibility of study design - exploratory studies

Answer 26

Begin with exploratory studies -> designing hypotheses -> pilot study -> test in real study

Question 27

When are eta-squared and partial eta squared equal?

Answer 27

When there is only 1 IV; when this is the case, there are no other factors that need to be “excluded” in partial eta squared.

Question 28

Why is (partial) eta squared not suitable to compare different studies?

Answer 28

Because the SSerror is different across studies, therefore giving different results