Hypothesis test Flashcards
Determine H & H0.
Are these outcomes equally likely?
H: are the counts /frq. of each category as expected?
H0: the probability of each group is exactly equal (or equal)
Determine H & H0.
are groups different from each other?
H: are the means of X different in the two groups
H0: the mean of Xa equals the mean of Xb
Determine H & H0.
did the outcome change?
H:did the mean of outcome X change across the two measurements?
H0: the mean of X1 equals the mean of X2
Determine H & H0.
are these two outcomes correlated?
H: are they the same?
H0: X and Y are uncorrelated (orthogonal)
Determine H & H0.
are these variables related
H: is there a linear relationship between X and Y
H0: the coefficient of alpha in an OLS regression of Y on X (and Z…) is zero
Procedure for a statistical test
1-based on theory + H0 there is a stat of interest that you can calculate with sample
2-use theory to derive distribution of expected values (under H0). some assumption must be made
3- calculate the statistic actual value given sample
4-state likelihood of answer from sampling method, given that the H0 is true, if unlikely = reject H0
What is the null hypothesis (H0)?
-what you are trying to disprove
- strong results, eliminate the possibility of the H0
- no difference, no change, small difference, no effect
reject the null if ( statistical significance)
p-value is small or near zero
type I error for binary decisions
rejecting the null hypothesis when it is true ( false positive)
type ii error
failing to reject the null hypothesis when it is false (false negative)
the probability of a type I error is determined by ____
significance level
- with a 99% significance threshold, type I error is less likely than with a 95% confidence
the probability of a type ii can be computed for a particular test statistic (i.e. H0), if given______
population distribution (parameterization, mean and SD)
sample size (N) and alpha ( chance of type I error)
hypothesis test power
1 - ( the probability of a type ii error)
AKA - the probability that we correctly reject the null, when the null is false
frequentist statistics
developed before computers & calculators ( t-test, chi-square, F test)
bayesian statistics
rapidly developing framework for using prior expectation + evidence to make bets
measurement validity
how well your metric captures the underlying concept you are trying to measure
internal validity
the degree to which the design of an experiment controls extraneous variables
external validity
the degree to which effects found in an experiment generalize to other individuals, contexts, and outcomes
(ex: in sampling studies, are the times and places outside of the sampling frame represented accurately)
Threats to external validity
interaction of selection and treatment
(unrepresentative responsiveness of treated pop.)
Threats to external validity
interaction of setting and treatment
effect of the treatment may differ across geographic or institutional settings
Threats to external validity
interaction of history and treatment
effect of the treatment may differ across time periods
Threats to external validity
effect may not persist
-as individuals and institutions adapt over time to the treatment
Threats to external validity
Partial- Equilibrium Effect
other components of the system also undergo related changes, reducing or eliminating the effect
