L9 - Statistical Power 2

Question 1

How would we plan an a priori power analysis?

(4 steps)

Answer 1

Use the relationship of sample size (N), effect size (ES), and our specified alpha (α) to calculate how many participants we need:

Decide on your α-level (Type I error rate)

Decide an acceptable level of power (Type II error rate)

Figure out the effect size (ES) you are looking for

Calculate N

Question 2

Why would we not set our level of power at .99 when designing our study?

Answer 2

ur alpha (α) level is fixed, and our effect size (ES) is fixed, which means we would need to have an incredibly large sample size (N)

the higher our desired power level, the more participants we need

Question 3

If we want higher power and have fixed effect size and fixed alpha, what would this mean for our sample size (N)?

Answer 3

We would need a larger sample size

Question 4

We don’t actually know the effect size before we run the study, so how can we determine the effect size before we run the study? (3 ways)

Answer 4

1. Base it on substantive knowledge

i.e. What you know about the situation and scale of measurement

2. Base it on previous research

What have others in your field used?

3. Use conventions

i.e. what is usually used

Question 5

What is the difference between a two-tailed and a one-tailed test?

Answer 5

Two-tailed, p value is split between the front and back (2.5-2.5)

One-tailed, only looking at significance in one way (5%)

Question 6

If we are doing a 2 tailed test, and the results are only just out of the significance range, can we change to a 1 tailed test post-hoc since we could have done so at the beginning anyway?

Answer 6

No, it is bad science and a type of p-hacking

Question 7

Cohen believes that an acceptable level of power is ___. This means we are happy to accept a type 2 error rate of ___.

This is a convention for an acceptable level of power that many use.

Answer 7

Power = .8

Type 2 = .2

Question 8

Which of these is the null distribution and which is the alternative distribution and why?

Answer 8

Null is on the left, for the significant range is in the (alpha ) zone

Alternative is on the right, because the type 2 error zone is denoted by ß

Question 9

What 3 statistics do we need to know before we can conduct a post hoc power analysis?

Answer 9

1. What was the sample size?
2. What was the a level (type 1 error rate)?
3. What was the effect size (ES)
   * With these 3, we can determine the power of the study.*
   * Example in notes and W9 powerpoint*

Question 10

When writing up your discussion and you didn’t have enough power, should you write that

1. You needed more power or
2. How big a study you would need to have acceptable power?

Answer 10

2. How big a study you would need to have in order to have acceptable power.

Question 11

What does this graph tell us about the level of power and the correlation significance?

Answer 11

There is an assumed power of .8

With a sample size of 20, we can detect correlations of about .6

40 of about .4

etc.

Question 12

It is really important to think of statistical power in terms of the ____ to the things that determine it.

Answer 12

Relationships

Question 13

The more stringent the significance level the ____ the necessary sample size

Answer 13

Greater

All other figures being equal

Question 14

The smaller the effect size the ____ the necessary sample size

Answer 14

Larger

Question 15

Why do we need bigger samples to detect smaller effects?

Answer 15

Because each of the 4 stats (N, ES, alpha, power) are related to power

• If power is constant at .8, alpha constant at .05, specify a small effect, that means we need a bigger sample than if we specified a big effect.*
• It’s an “all else being equal scenario” as all things are related to each other.*
• Important I think*

Question 16

The higher the power required the ___ the necessary sample size

Answer 16

Larger

Question 17

The smaller the sample size the ___ the power

Answer 17

Lower

Question 18

What is the risk of having journals that demand statistically significant results and low powered studies?

Answer 18

Articles that gets published being type 1 errors

This is the replication crisis

Question 19

What are the 4 relationships that are all related to each other in power analysis?

Answer 19

Power

Effect Size

Sample Size

Significance level

Question 20

What is power analysis good for?

Answer 20

1. Sample size calculation

Before you begin your study

2. Evaluation of study results

• Published literature*
• • Useful when planning studies – ES estimates*
• • Replicability?*
• Your own results*
• • What power did your study have (be careful here)*
• • Given the observed ES, what size study to reliably detect? (more useful)*
• Was my study too small? Or too large?*
• • Ethical implications?*

Question 21

Why can having a sample size that is too small be considered unethical?

Answer 21

A too small sample size means that you will never have enough power for your results to be relevent.

This means you have wasted the participants time, unethical.

Question 22

One limitation of power analysis is that it uses the frequentist NHST decision making framework.

Why is this a limitation?

Answer 22

If you’re using power analysis outside this framework, none of it makes sense

Question 23

The concepts of Type I and Type II error at the heart of power analysis depend on what many consider to be the pathological nature of NHST.

Why is this so?

Answer 23

Because we are basing it off an arbitrary level of alpha. This is an arbitrary cutoff point.

• Why should we treat p = .051 differently to p = .049?*
• One implies that one study is meaningful, whereas the other is meaningless.*

Question 24

What is more important for significance, effect size or p values?

Answer 24

Effect size

P values are arbitrary - whereas effect size is much more meaningful, particularly when standardised

Question 25

What is the weakness of using the decision making NHST model?

Answer 25

It was meant for quality control in industry, not necessarily scientific modelling

Just because our study meets the alpha level, doesn’t necessarily mean that we should reject the null, as alpha is arbitrary.

Conversly, just because we didn’t find significant results, doesn’t necessarily mean that we’re certain it doesn’t have any effect.

Question 26

When we find significant or non-significant results in a power analysis, do we need to make a decision on whether our results are accurate?

Answer 26

No

All we need to do is report our results, then we can let others continue to build on those results in order to determine if it is accurate

Question 27

According to Burns, are 1-tailed p value statistical tests illogical?

Answer 27

Yes.

It changes nothing, because it is only making it more likely to be significant based on the arbitrary p-value, meaning you are kidding yourself if it is only just in the p <.05 range.

The effect size hasn’t changed, only how you view the effect size.

Question 28

Is statistical significance the same as scientific significance?

Answer 28

No

Just because something is statistically significant based on a arbitrary p value, doesn’t necessarily mean it’s scientifically valid.

Question 29

A non-significant result means that we should always retain the null.

Answer 29

Not necessarily

Instead, suspend judgements, because it could be a type 2 error.

Question 30

Rejecting H0 because we found significant results means that our hypothesis is correct.

(True or False)

Answer 30

False

A positive difference, a negative difference, or that the direction of the difference cannot be confidently stated

Question 31

If we have statistically significant results and reject the null, we can confidently say which direction the results were in.

Answer 31

Not necessarily.

Rejecting H₀ implies either a positive difference, or a negative difference. The direction of the difference cannot be confidently stated.