Week 5 - Statistical Power Analysis Flashcards
Learning Objectives: Understand Null Hypothesis significance testing Identify Type I and Type II errors Understand how alpha, sample size, and power are interrelated What are some methods that are used to reduce Type I error How is power associated with Type II error Advantages and disadvantages of Open Science and pre-registration What is an effect size? How are Confidence Intervals useful? How can we calculate sample sizes required for our study?
what is Null Hypothesis Statistical Testing (NHST)
- H0 (null hypothesis): The two groups (experimental vs. control) are not different on the dependent variable.
- H1 (alternative hypothesis): The two groups are different on the dependent variable.
We design our study to determine whether we can “reject” the null hypothesis.
what is type 1 error?
We reject the null hypothesis when it is in fact true (groups are really not different on the dependent variable but we think they are).
- we might advocate for a treatment that doesn’t really work
“I falsely think the hypothesis is true” (one false)
what is type 2 error?
We fail to reject the null hypothesis when it is in fact false (groups really are different on the dependent variable but we conclude they are not).
- we might neglect a treatment that does work.
“I falsely think the hypothesis is false” (two falses)
what is a two group comparison?
what is the alpha level?
Alpha (α) is the probability of making a Type I error (rejecting the null hypothesis when we shouldn’t). We only want to do this rarely – 5 times out of 100 when the two groups are not really different is the usual cut-off.
by tradition, statistical significance has been determined at p < .05. The p value is the probability that you would find the current result if the two groups really weren’t different.
what is the credibility lab?
a way to preregister for your study
what is preregistration
design plan - items are date stamped so when you write your publication, theories were created before finding results
somewhat prevents harking
what is the centre for open science
supports open science
mission to “increase the openness, integrity, and reproducibility of scientific research
what is the replication crisis?
when researchers find similar studies and replicate in order to
led to scientists engaging in more open science practice
the growing belief that the results of many scientific studies cannot be reproduced and are thus likely to be wrong.
why do some adjust the alpha levels?
- Researchers occasionally try to make the case for a higher alpha level (p < .10, for example). Or more often, they call these effects “marginal” or a trend. However, this is just increasing the likelihood that a Type I error has been made (treating a group difference as real when it is not).
- P values do not tell us anything about the size or magnitude of an effect – they just tell us, dichotomously, about whether it is likely if the groups really weren’t different.
- When we conduct a large number of tests, particularly exploratory tests, it is traditional to reduce your alpha levels to be more cautious about making a Type I error.
- Bonferroni correction divides the alpha level (.05) by the number of tests. A priori theory based predictions are not as risky. Also see Benjamini-
Hochberg procedure for controlling for False Discovery Rate
What is p-hacking?
exploitation of data analysis in order to discover patterns which would be presented as statistically significant, when in reality, there is no underlying effect.
what is experimental power and the type 2 error?
Beta (β) is the probability of making a Type II error (declaring a real difference between groups not to be there).
Power = 1 – β
In other words, the more you can reduce the likelihood of missing a real difference, the more power you will have to find effects.
power is higher when effects are larger (easier to find) – when the real difference between groups is very large. Moreover, because any given sampled group is normally distributed around the population mean, having a larger sample ensures a more accurate assessment of that group and thus more power/confidence that we have found a real effect. Small effects require larger samples in order to see the real group difference beyond sampling error.
what are effect sizes?
Effect size, the amount of something of interest An effect size (ES) can be:
▪ A mean, or difference between means
▪ A percentage, or percentage change
▪ A correlation (e.g., Pearson r)
▪ Proportion of variance explained (R2, w2, h2…)
▪ A standardised measure (Cohen’s d, Hedges g…)
▪ A regression slope (b or b)
▪ A measure of goodness of fit
▪ Many other things… (but NOT a p value!)
Kelley, K., & Preacher, K. J. (2012)
what is cohen’s d?
one of the most common ways to measure effect size. An effect size is how large an effect is. For example, medication A has a larger effect than medication B
d = 0.2 be considered a ‘small’ effect size, 0.5 represents a ‘medium’ effect size and 0.8 a ‘large’ effect size. This means that if the difference between two groups’ means is less than 0.2 standard deviations, the difference is negligible, even if it is statistically significant.
what is a 95% confidence interval (CI)
a range of values that you can be 95% certain contains the true mean of the population.
