WEEK #4(b) - research methods Flashcards
what is significance testing based on ?
the statistical properties of sample data
what can we extrapolate “estimate” with significance testing ?
the probability of the observed differences or relationships occurring in the target population
what are we assuming in regards to significance testing ?
that the sample data is representative and that that data meets the assumptions associated with the inferential test
who developed significance testing ?
Ronald Fisher (1920s-1930s)
TRUE OR FALSE
agricultural research designs couldn’t be fully experimental because natural variations such as weather and soil quality couldn’t be fully controlled
TRUE
what are some criticisms of significance testing ?
- the null hypothesis is rarely true
- no clinical effect / no change
what does significance testing provide :
- a binary decision (yes or no)
- direction of the effect
what does statistical significance simply mean ?
that the observed effect (relationship or differences) are unlikely to be due to sampling error
TRUE OR FALSE
statistical significance can be evident for very small (trivial) effects if N and/or critical alpha are large enough
TRUE
what does level of significance (aka alpha) represent ?
the probability of obtaining your results due to chance
the smaller the value is, what does this mean for level of significance (alpha) ?
the more “unusual” the results, indicting that the sample is from a different population than its being compared to (for example)
what happens if p-values fall below significance level for statistical significance alpha ?
we say that the results from the test are statistically significant
what happens if p > a ?
then FAIL TO REJECT the null hypothesis
what happens if p < a ?
then REJECT the null hypothesis
TRUE OR FALSE
alpha does not represent your chance of making a type 1 error
FALSE
alpha DOES represent your chance of making a type 1 error
define power beta :
refers to your study’s ability to find a difference if there is one
the greater the power, what does this mean ?
the more meaningful your results are
what does beta = ?
1 - power
FILL IN THE BLANK
beta represents the chance of making a __________
type 2 error
in regards to power beta what do you incorrectly fail to reject ?
the null
what is the desirable power of beta ?
desirable power > .80
what is the typical power of power beta ?
typical power around 60
when does power become higher ?
when there is an increase in sample size (N), critical alpha (a), and effect size (delta)
based on what can we calculate expected power before conducting a study ( a priori) :
- estimated N
- critical alpha
- expected or minimum effect size )from related research)
what is effect size ?
a measure of the strength (size) of a relationship or effect
what is an inferential test ?
used as a way to quantify if there is an association, a difference or relationship between variables
statistical tests used to determine whether the research results can be generalised to the general populatio
TRUE OR FALSE
an inferential test may be statistically significant, but this doesn’t indicate hw large the effect is
TRUE
are effect sizes influenced by N ?
NO
unlike significance testing, effect sizes are not influenced by N
is significance testing influenced by N ?
YES
what is a common effect size statistic ?
Cohen’s d
what is Cohen’s d ?
equal to the mean difference divided by the standard deviation
when using Cohen’s d what are the three effect sizes ?
d = 0.2 small effect
d = 0.5 medium effect
d = 0.8 larger effect
according to Cohen’s d, what is a small effect ?
d = 0.2
according to Cohen’s d, what is a large effect ?
d = 0.8
according to Cohen’s d, what is a medium effect ?
d = 0.5
TRUE OR FALSE
p-value is the exact level
TRUE
FILL IN THE BLANK
alpha is the maximum ___________ (max level)
type 1 error
TRUE OR FALSE
type 1 error is the probability of type 1 error
TRUE
what is type 1 error the probability of ?
rejecting a correct null hypothesis
define a small p-value :
- small enough p-values gives us reason t reject H0 and support HA
what do p-values tell us exactly ?
how likely we are to make a type 1 error if we reject H0
for p-values, what size is better ?
smaller is better (in support of alternative hypothesis)
(in plain English) what is the p-value ?
the probability of incorrectly rejecting the null hypothesis OR the probability of rejecting a null hypothesis when in fact it is “true”
FILL IN THE BLANK
_________ is the chance of error you will have to accept if you want to reject the null hypothesis
the p-value
if a p-value is 0.01 what does this mean ?
there is a 1% chance that we will incorrectly reject the null hypothesis, or that we could reject the null hypothesis with a 1% chance of error
if a p-value is 0.04 what does this mean ?
there is a 4% chance that we will incorrectly reject the null hypothesis, or that we could reject the null hypothesis with a 4% chance of error
if the p-value is between 0 and 0.01, what does this imply ?
overwhelming evidence
if the p-value is between 0.01 and 0.05, what does this imply
strong evidence
if the p-value is between 0.05 and 0.10, what does this imply ?
weak evidence
if the p-value is greater than 0.10, what does this mean ?
no evidence in favour of HA
when talking about alpha and p how should you think of the alpha ?
think of the alpha as the minimum p-value you are willing to accept
what are the three steps of alpha and p ?
1) set the alpha level before you calculate the p value
2) calculate the p-value
3) compare the p-value to the alpha level
if p > a what does this mean ?
then FAIL TO REJECT the null hypothesis
if p < a what does this mean ?
then REJECT the null hypothesis
what are the two ways in which p-values can be reported ?
- actual p-values (P = 0.04)
- statement on inequality (P < 0.05)
TRUE OR FALSE
most of the time, the null effect is true
FALSE
most of the time, the null effect is NOT true (impossible)
what is the only way we can calculate probability for p-values misconceptions ?
Bayes Theorum
what is the p value in null hypothesis significant testing condition on ?
the null hypothesis being true
what does “the p value in null hypothesis significant testing condition on the null hypothesis being true” mean ?
that a p value of 0.05 does not mean that the probability our data arose by chance alone is 1 in 20
FILL IN THE BLANK
the chance of us mistakenly rejecting the null hypothesis and concluding we have a successful treatment is more in the region of _________ %
30 - 60 %
what is a p value problem ?
American statistical association argued, “by itself, a p value does not provide a good measure of evidence regarding a model hypothesis”
FILL IN THE BLANK
the false positive rate associated with a p-value of .05 is usually around _____, but can be much higher
30%
even when a p-value is interpreted correctly, what is it silent on ?
the magnitude an range of an effect
