Week 5 - Power and Effect sizes Flashcards
When we test a hypothesis we…
Start by assuming the null hypothesis is true and then let the p-value tell us how likely we would get the data that we observed.
How to decide if something is true?
- If the p-value is low (less than .05), it suggests that the observed data is unlikely under the nullhypothesis, so we reject the null hypothesis in favour of the alternative hypothesis.
- If the p-value is high, it means that the observed data is plausible under the null hypothesis, so we do
not reject the null hypothesis. - This process helps us determine whether there is enough evidence to reject the null hypothesis and
support the alternative hypothesis.
If the null hypothesis is in fact true, but we have a p-value of p=.002 we have…
made a type I error
If we set our threshold for significance at .05, and a
group of students run an experiment where the null hypothesis is true, what percentage of students are likely going to make a Type I error:
5%
Power is…
The probability that our experiment will find a signficiant result, if the effect is real.
- Statistical Power is the probability that our experiment will find a significant result, provided that the effect we are testing for actually exists. Higher power means a higher probability of detecting an effect when there is one. Typically, researchers aim for a power of 0.8 or 80%, which indicates a good chance of detecting an effect if it exists.
- This concept is important for experimental design because it influences the sample size and the interpretation of results. Ensuring sufficient power can help researchers avoid Type II errors, where we fail to detect an effect that actually exists.
What is the null effect?
- When we see a uniform distribution of p-values, it
suggests that there is no systematic effect present in the data. This is because a true null hypothesis would result in p-values being evenly distributed between 0 and 1, indicating that any observed significant results are likely due to random chance rather than a real effect.
-In other words, if the null hypothesis is true and we run the experiment many times, the p-value should be uniformly distributed because there’s no underlying effect influencing the data.
what do P-Curve skews indicate
- When the p-curve is skewed towards lower p-values, it
suggests that there are consistent significant effects being
found in the studies. This pattern implies that the variable
being tested (in this case, the emotional content of stimuli) has a real and meaningful impact on the outcome (our ability to remember them). - Essentially, a p-curve with most values at the lower end
indicates strong evidence against the null hypothesis and
supports the alternative hypothesis that emotional content plays a role in memory retention.
If you and a friend run separate experiments, each
with 50% power, and H1 is true, what is the probability
that you will both find a significant effect…..
25%
Power of an Experiment:
Power (typically denoted as 1 - β) is the probability that
an experiment will correctly reject the null hypothesis when the alternative hypothesis (H₁) is true. In this case, each experiment has a power of 50%, or 0.5.
Independent Events: When you and your friend run separate experiments, each with 50% power, the outcomes of these experiments are independent events.
Calculating Joint Probability: To find the probability that both of you will find a significant effect, we multiply the probabilities of each event occurring. So, the
probability that both experiments will find a significant effect is:
.5 *.5 = .25
So there is a 25% chance that both experiments will result in finding a significant
effect when H₁ is true
Definitions
Sample Size and Statistical Power: Statistical power (1 - β) is the probability that a test will correctly reject the null hypothesis when the alternative hypothesis (H₁) is true. Increasing the sample size reduces the standard error, leading to more precise estimates of the population parameter.
Type II Error (β): A Type II error occurs when we fail to reject the null hypothesis when it is actually false. Increasing the sample size decreases the probability of making a Type II error, thus increasing the statistical power.
Type I Error (α): The probability of a Type I error (incorrectly rejecting the null hypothesis when it is true) is determined by the significance level (α) and is not
directly affected by sample size. It remains constant regardless of sample size.
The higher your t-value…
The lower your p-value
T-Value and P-Value Relationship: The t-value is a statistic that results from the t-test. It measures how much the sample data deviates from the null hypothesis, that is the size of the difference between conditions, relative to the variation in your sample data. The p-value is a probability that measures the evidence against the null hypothesis. (The p-value measures the probability of obtaining test results at least as extreme as the observed
results, assuming that the null hypothesis is true, that is how likely we are to find these results purely by chance, if there truly is no effect).
A higher t-value indicates that the observed data is further away from what we would expect under the null hypothesis, suggesting stronger evidence against the null
hypothesis.
Consequently, a higher t-value corresponds to a lower p-value, indicating that the probability of observing such data (if the null hypothesis were true) is very low.
If the alternative hypothesis is true, what can we do to
increase the power of our study:
A. Increase the sample size (the number of participants we are testing)
