W10 - P-Values and NHSTs Flashcards
In a trial-by-jury: What is the null hypothesis analogous to
Presumption of innocence
In a trial-by-jury: What is the alternative hypothesis analogous to
Commiting the Crime
In a trial-by-jury: What is the alpha value analogous to
Beyond a reasonable doubt
In a trial-by-jury: What is the trial evidence analogous to. How is it collected?
- p*-value
- Collected by data and expressed as a p - value from the relevant probability distribution under the assumption of the null hypothesis (Central distribution?)
Student’s t50 = 2.403, p = .02
In a trial-by-jury: What is the comparision between p-value and alpha value analogous to. Can p = .05?
Comparision of trial evidence and beyond a reasonable doubt criterion
- note: obtained p will never be precisely .05 with continous distributions
- If p < alpha, the null is rejected in favour of the alternative hypothesis. If p > alpha, the null is not rejected.
In a trial-by-jury: What is the decision analogous to
Guilty: Rejecting the Null Hypothesis
Not Guiltry: Not Rejecting the Null Hypotehsis
In a trial-by-jury: What is the a type 1 error analgous to
Type 1 error: p < alpha, but the null is true
- The person is found guilty but did not commit the crime
In a trial-by-jury: What is the a type 2 error analgous to
Type 2 error: p > alpha, but the null is false
- The person is not found guilty but did commit the crime
Do we know the Null Hypothesis is true/not in practice?
No.
But we act in terms of it being true/false based on evidence provided by the data.
What are null hypothesized value
One value that we are proposing for the unknown population parameter
It does not have to be zero.
We use tests to examine how our sample statistic compares to this proposed population parameter
Define p value and its equation
- p-*value is the conditional probability of the sample effect size being observed or one larger, given the null hypothesis is true.
- p* = Pr (Tobs | H0 = True)
- Tobs = Effect size or one larger
What is the observed test statistic value for a student’s t
Student’s tobs = [(M1 - M2) - (μ1 - μ2)] / SEM1-M2
- M1 - M2
- Difference in means between group 1 and group 2
- μ1 - μ2
- Null hypothesis size difference. While this is mostly 0, it can be non-zero.
- SEM1-M2
- Standard error of mean difference
What does “…or one larger” suggest about the p values
P Values contained
- Estimated effect size observed in the data
- Larger-sized effects that are not observed in the data
- Both features are required to be able to calculate the P-Value
- P-Values contain more than what is observed in the data…
Define Alpha Value and its equation. In other words…
Alpha value is the conditional probability of rejecting the null hypothesis, given that the null hypothesis is true
a = Pr (Rejecting H0 | H0 = True)
In other words.. alpha value is the probability of type 1 error IF h0 = true
- Both alpha and p value are conditional that H0 = true
- But alpha is based on an action (Rejecting H0) but p-value is based on observed data (Tobs )
What does alpha value set to achieve.
# * Define the _maximum_ chance of _falsely rejecting_ a _**true** null hypothesis_ over the long run when _all statistical assumptions hold_ * Controlling how much we are wrong
Alpha is NOT type 1 error rate. It is the probability of type 1 error IF the null hypothesis is in fact true!
If alpha = .05, what does it mean
Over a large number of independent samples, the probability of false rejecting a true null hypothesis will be 5% over the long run.
i.e. commiting a type 1 error IF the null hypothesis is true
Define type 1 error and its equation
Type 1 error is the conditional probability that the null hypothesis is true given that the null hypothesis is rejected
- Pr (H0 = True | Rejected H0)
How can we best distinguish Type-1 Error and Alpha Value?
-
Alpha Value
- Pr (Rejecting H0 | H0 = True)
- Set before analysis
- Value of alpha is unchanged to what it was set to regardless of Type 1 Error
-
Type 1 Error
- Pr (H0 = True | Rejected H0)
- Occur after analysis and a decision is made to reject the null hypothesis.
- If null is not rejected, probability of type 1 error = 0
- If null is not true, probability of type 1 error = 0
If the null hypothesis is true, what will be the distribution of p values
It will be uniform
(each p-value will be an equal probability of being obtained)
If the null hypothesis is false, what is the distribution of p-values
It will not be uniform. Might result in high type 2 error.
How would I increase my rejection rate?
(If I somehow knew that H0 = False)
- Increase sample size
- Increase diference at a population level
What is the meaning of a p-value (in relation to population paramter)
Measure of consistency/compatability of sample data with the null-hypothesized population parameter.
The smaller the p, the less consistent the data with the null hypothesis…
What does the binary decision to reject/not reject do?
Places a boundary on the frequency of making an incorrect rejection of a true null hypothesis over the long run when all statistical and research design assumptions have not been violated
What are factors affecting the p-value
- Actual effect size/true state of affairs
- Statistical assumptions
- Research design assumptions (such as sample size)
What are the misintepretations of p-values discussed in the lecture?
- Tells us null hypothesis is true or false
- Observed effect is due to chance alone
- p>alpha means no effect
Misinterpretation of p-value 1: Tells us null hypothesis is true or false. Elaborate
We cannot use p to make any probabilisitc statement about the truth/falsity of the null hypothesis.
To do so, we require Pr (H0 = True | Tobs)
However,
Pr (Tobs | H0 = true) /=/ Pr (H0 = True | Tobs)
Misintepretation of p-value 2: The observed effect is due to chance alone
p-value: Both assumed null hypothesis plus all statistical and research design assumptions
- P-value is calculated on assumption that chance alone is operation
- Does not indicate the probability of chance being the explanation (i.e. could be assumptions)
Misintepretation of p-value 3: P > alpha = no effect is present
There are large number of null-hypothesized values for the population paramter than can result in P > alpha in a single sample
It does not tell us the true state of affairs.
Only circumstance which we can say there is no effect present will only occur if sample statistic is EXACTLY equal to null hypothesized values, which in practice will never occur.
What is a p-value function plot
Plot of obtained p values against null hypothesized values
If the obtained P-value = 1, what does it mean in a p-value function plot. How does the p-value change as a function of null hypothesized change
- It is the peak in the p-value function plot
- It is when the
- Null hypothesized value = Sample Statistic
- Comptabile (Not rejected)
- As null hypothesized values move away, it is has lower p values
- Imcompatible (Rejected)
- And thus it is in that sense CI is the range of values that would NOT be rejected
- Null hypothesized value = Sample Statistic
What does the bounds of the confidence interval tell us
It tell us the values in which p = .05
Define confidence interval in the sense of p-values
Confidence intervals calculated in a single sample defines the complete set of null hypothesized values that would NOT be rejected if used in a NHST test on a sample statistic.
- Thus, we can regard a CI as a set of plausible values for the unkown population parameter value in that they are null hypothesized values that would not have been rejected
What are the misinterpretations of a confidence interval discussed in the lecture
- It has a 95% of capturing the true effect size
- Any value outside the interval are population effect that are ruled out
Misinterpretation of confidence interval 1: It has a 95% of capturing the true effect size
- Any single confidence value either (a) Capture unknown population parameter (b) Does not
- Over the long run, 95% of all confidence intervals calculated from independently-replicated samples will contain the true effect size
Misinterpretation of confidence interval 2: Any value outside the interval are population effects that are ruled out
- Values not captured = Null-hypothesised values that would be rejected in a NHST
- Does not mean values can DEFINITELY ruled out as population effect sizes
- Does not mean values outside the interval have a 5% chance of being true
What is the formula for lower and upper bound of CI. How does sample size and variability relate to it?
Bounds
- θobs +- (TCrit x σSE )
- Since σSE = (σ / sqrt n)
- As σ increases, CI width increases, less precise
- As n increases, CI width decreases, more precise
- Since σSE = (σ / sqrt n)
Define Coverage Rate in Confidence Intervals
Proportion of the time that the confidence interval contains the true populaton parameter value
If h0 = true, as n increases, what happens to rejection rate, CI coverage rate, and confidence intervals
Rejection rate
- Same for each sample size
CI coverage rate
- Same for each sample size
Precision of CI (Width)
- Narrower Width
- More precise confidence intervals on average
If h0 = false, as n increases, what happens to rejection rate, CI coverage rate, and confidence intervals
Rejection rate
- Increases as n increases
- Note: Rejecting is a correct decision
CI coverage rate
- Same for each sample size
Precision of CI (Width)
- Narrower Width
- More precise confidence intervals on average
If h0 = false, as n increases AND effect is larger, what happens to rejection rate, CI coverage rate, and confidence intervals
Rejection rate
- Increases as n increases
- Note: Rejecting is a correct decision
- Larger effect leads to larger rejection
CI coverage rate
- Same for each sample size
Precision of CI (Width)
- Narrower Width
- More precise confidence intervals on average
- No difference of effect
If h0 = true, if n is balanced and homogenegity is violated, what happens to rejection rate, CI coverage rate, and confidence intervals
Rejection rate
- Does not change with variability
- As expected
CI coverage rate
- Does not change with variability
- As expected
Precision of CI (Width)
- Greater variability = Less precise CI (Wider width)
- Smaller variabilty = More precise CI (Narrow width)
If h0 = true, if n is unbalanced and homogenegity is met, what happens to rejection rate, CI coverage rate, and confidence intervals
Rejection rate
- Does not change with variability
- As expected
CI coverage rate
- Does not change with variability
- As expected
Precision of CI (Width)
- As expected
If h0 = true, if n is unbalanced and homogenegity is violated, How do we know the direction of skew? How does it affect rejection and coverage rates?
Group 1
- Smaller Sample
- Smaller Variance
- Negative Skew
- Rejection Rate: Smaller than expected (<5%)
- Coverage Rate: Larger than expected (>95%)
- Negative Skew
Group 2
- Smaller Sample
- Larger Variance
- Positive Skew
- Rejection Rate: Larger than expected (>5%)
- Coverage Rate: Smaller than expected (<95%)
- Positive Skew
By defintion, since interval estimator does not equal expected rate defined by level of confidence, it is biased
If h0 = true, if n is unbalanced and homogenegity is violated, what happens to rejection rate, CI coverage rate, and confidence intervals
Positvely or Negatively skewed
Rejection rate
- Much smaller or larger
CI coverage rate
- Much smaller or larger
Precision of CI (Width)
- Wider or Narrow (Depends on variability and sampling size)
