L5 Flashcards
Are p values good at estimating magnitude?
No, only significance
In psychology, when should we reject the null hypothesis?
p < .05
Name the 4 types possible outcomes of a test using signal detection theory
Hit
False Alarm
Miss
Correct Rejection
What is it called when you want to decrease the number of type 1 errors in your experiment?
Conservatism
How might you make your experiment more conservative?
Decrease the response criterion to lower than p <.05
Why might you not want to make an experiment more conservative?
Because you increase the number of ‘miss errors’ and the tradeoff seems to be non-linear
(Much more likely to get ‘type 2 errors’ if more conservative)
Seemingly small shifts in our response criterion can decrease the sensitivity of our experiment because we don’t know how often genuine effects happen in reality.
What is the difference between the null hypothesis and the research (alternative) hypothesis?
When we run a significance test, we are measuring how likely it is that our effect has come from the ‘chance distribution’ instead of the alternative distribution.
If we get a mean of 3 in our sample data below, how do we determine if it is from the chance or the alternative distribution?
Does the p-value tell us the
probability of the data given the hypothesis
or
the probability of the hypothesis given the data
probability of the data given the hypothesis
What is the 5% cutoff (.05) we use to reject the null hypothesis called?
The p-value is defined as -
The probability of getting an effect at least as big as the one we got if the null hypothesis is true
With a alpha of .05, how often would we get a type 1 error?
5%
Do significance tests tell us the probability that the effect exists in the wild?
What are these tests called?
No
This is why they are called inferential tests
We are inferring that our small sample of observations is representative of the population
Significance tests give us the probability of the data, given the null hypothesis that our sample came from a ‘chance distribution’
True or False
True
p-values are a good measure of the magnitude of the effect
True or False
False
They can only tell us if the effect is significant.
If there is no effect, what will the distribution of p-values look like?
Spread out evenly
How might we see if researchers have manipulated the results to get just under the significance of p < .05?
What is this called?
Analyse the range of p-values for the results, if there is a spike at just under .05 the results has likely been manipulated
This is called a p-curve analysis
(red line in example graph is sign of manipulation. Top graph is no difference in the groups (null is true), bottom is a genuine difference, with more trending towards 0)
What is an effect size?
A measure how big a difference is or how strong a relationship is
Is it better to use effect size than p values?
Yes, p values can give you an estimate of whether there is a significant difference but it can’t tell you the magnitude (size) of the effect
What is a raw effect size dependent on?
Dependent on the scale or the measure being used in the experiment.
What are the disadvantages of using the raw effect size?
- You need to be familiar with the DV and scales to make sense of the size of the effect and whether it is big compared to other experiments in the field
- When we look at studies overall, it is difficult to estimate the contribution of a particular variable because we are comparing apples and oranges.
* In order to compare across studies we need to compare on the same scale*
Why are standardised effect sizes useful?
Indepenndent of the scale (DV) being used in the experiment.
Therefore you can compare effect sizes across different experiments. (Good for meta-analysis)
What type of effect size is Cohens D?
Standardised effect size
What is the formula for Cohen’s d
the difference between two group means of the two levels of IV of interest, divided by the population SD
We don’t know the population SD actually, so instead of population SD we compute the pooled standard deviation (SDpooled)
What is a small, medium, large Cohen’s D effect size?
Small <.2, Medium <.5, Large >.8
Somewhat arbitrary though
What is the standardised effect size of Correlational research?
Pearson’s r
What does Pearson’s r tell us?
The strength of the relationship between two paired variables
What are the Pearson’s R interpretation conventions?
Small < .10
Medium < .3
Large > .5
What is statistical power?
The probability of rejecting the null hypothesis if the null hypothesis is false (1 - ß)
What is statistical power dependent on?
1) Number of observations
2) Precision
3) Effect size
What is the formula for statistical power?
1 - B (beta, the hit rate)
Increasing statistical power is the same thing as reducing ___
ß
What are 8 things you can do to increase your chance of detecting an effect when it exists in reality?
Most of the things that you can alter to increase your chance of detecting an effect occurs at what stage of the research process?
Design stage
What power do most people believe should we aim for in an experiment?
80% power
1-ß = .8
We usually want to estimate statistical power before we do our experiment.
What is the drawback of doing this?
We might not have a good idea of the effect size we are looking for (if we had an idea, we probably wouldn’t be running the test in the first place)
What calculation can we use to estimate our effect size beforehand?
Smallest Effect Size of Interest (SESOI)
With statistical power, if the null hypothesis is false, what are the two possible outcomes that equal 100%
Make a mistake and fail to reject it (probability = ß)
Correctly reject the null (probability = 1 - ß)
Total probability = 100%
Increasing statistical power is the same thing as reducing ß
