Bayesian Flashcards
Write down Bayes theorem (simple case)
p(B|A) = \frac{p(A|B)p(B)}{p(A)}
Where
p(A) = \sum_{B_i \in S_B}p(A|B_i)p(B_i)
Describe the differnet components in the Bayesian formula
$p(B)$ is the marginal probability of B and is considered “prior” information since it exists prior to our situation. In the case of a pregnancy test “The “prior” information we need, $p(B) ≡ p(preg)$, is the marginal probability of being pregnant, not knowing anything beyond the fact that the woman has had a single sexual encounter. This information is considered prior information, because it is relevant information that exists prior to the test. We may know from previous research that, without any additional information (e.g., concerning date of the last menstrual cycle), the probability of conception for any single sexual encounter is approximately 15%”.
$p(B|A)$ is called the “posterior” probability. This is because it is the updated belief after observing the data.
$p(A|B)p(B)$ is called the “Kernel”
How do we go from the point-probability Bays thing to the probability distribution
Bayesian statistics typically involves using probability distributions rather than point probabilities for the quantities in the theorem. We thus replace the prior with a hole prior distribution which captures the uncertainty regarding the prior. The inclusion of a prior probability distribution ultimately produces a posterior probability that is also no longer a single quantity; instead, the posterior becomes a probability distribution as well.
Can you put probability distributions on parameters in the Bays perspective?
From the Bayesian perspective, any quantity for which the true value is uncertain, including model parameters, can be represented with probability distributions. From the classical perspective, however, it is unacceptable to place probability distributions on parameters, because parameters are assumed to be fixed quantities
Write the Bays formula and describe the parts in the case when we use probability distributions.
See notion.
where $f(θ|data)$ is the posterior distribution for the parameter $θ, f(data|θ)$ is the sampling density for the data—which is proportional to the Likelihood function, only differing by a constant that makes it a proper density function—$f(θ)$ is the prior distribution for the parameter, and $f(data)$ is the marginal probability of the data.
Because this denominator simply scales the posterior density to make it a proper density, and because the sampling density is proportional to the likelihood function, Bayes’ Theorem for probability distributions is often stated as:
$$
Posterior ∝ Likelihood × Prior
$$
This equation will be “off” by the denominator on the right side of the equation for the posterior above, in addition to whatever normalizing constant is needed to equalize the likelihood function and the sampling density $p(data | θ)$.
What is proportionality?
Specification of an appropriate prior distribution for a parameter is the most substantial aspect of a Bayesian analysis that differentiates it from a frequentist analysis. Using the example of pregnancy from ~~here~~, we could replace the point estimate of .15 with a probability distribution that represented the plausible values of the prior probability of pregnancy and their relative merit. For example, we may give considerable prior weight to the value .15 with diminishing weight to values of the prior probability that are far from .15. Similarly, in the polling data example, we can use a distribution to represent our prior knowledge and uncertainty regarding $\theta$
.
We like a distribution that insures $\theta\in[0,1]$
Describe the beta distribution and how we should set the parameters.
An appropriate prior distribution for an unknown such as $\theta$
is a beta distribution. The pdf of the beta distribution is:
$$
\frac{\Gamma (\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\theta^{\alpha-1}(1-\theta)^{\beta-1}
$$
With higher $\alpha$ and $\beta$ we get less variance in our prior distribution. If we have little or no prior information, or we want to put very little stock in the information we have, we can choose values for α and β that reduce the distribution to a uniform distribution, i.e., setting $\alpha = \beta = 1$. This type of prior is called “noninformative.” If we have considerable prior information and we want it to weigh heavily relative to the current data, we can use large values of α and β. If we have considerable prior information but we do not wish for it to weigh heavily in the posterior distribution, we can choose moderate values of the parameters that yield a mean that is consistent with the previous research but that also produce a variance around that mean that is broad.
With little data the choice of prior (the values of alpha and beta) will be of significant importance, but that is not the case if we have a lot of data.
What is an improper prior?
A probability distribution for $θ$, $p(θ)$, is called improper if its integral over the sample space does not converge. E.g., $\theta \in(-\infin,\infin)$. This seems to be a uniform distribution? That is, we have $\alpha = \beta = 1$
This is essentially a rectangle with an infinitely long base, so its area diverges. We give all events the probability of 1???
Reasons to use improper priors are:
- That they can be thought as an approximation to a proper prior representing very vague beliefs.
- A uniform prior is also a quick and convenient way to start doing the analysis
without worrying about specifying your beliefs.
What is the posterior?
The posterior density represents your beliefs about $θ$ given your prior beliefs
and the beliefs embodied in the likelihood. • It is the culmination of the empirical analysis. Equivalent in frequentist econometrics: table with estimates of $θ$ and its standard errors.
How do we think about the distribution of the posterior
The distribution of the posterior
$$
Posterior ∝ Likelihood × Prior
$$
We study this and guess what distribution it is. It might e.g., by that ir resembles a beta distribution. Then it is a beta distribution where e.g.,$\alpha = a+b$ and $\beta = c+d+e$. When the prior and likelihood are of such a form that the posterior distribution follows the same form as the prior, the prior and likelihood are said to be conjugate
How do we report the posterior?
In some cases (such as with conjugate priors), we know the posterior analytically. Sometimes we don’t know the posterior analytically, but we can draw samples
from it (using Monte Carlo simulations or importance sampling).
In Bayesian analysis, we can report the mean of the posterior and an interval where we can say that with, say, 95% probability we believe that the true parameter lies in that interval.
What is an credible interval?
With the posterior density determined, we now can summarize our updated knowledge about $\theta$. If we are willing to assume that this beta distribution is approximately normal, then we could construct a 95% interval based on a normal approximation. This interval is called a “credible interval”.
If we like to know the probability that $\theta$ is larger than $.5$, we integrate the beta distribution of this density from .5 to 1.
What is the likelihood in the Bays formula?
The likelihood $p(data|θ)$ is simply the probability of the data conditional on the parameter θ.
If the data follows a binomial distribution then the likelihood function is
What is the Bayesian algorithm
We can formulate the Bayesian method as an algorithm
- Formulate an economic model as a collection of probability distributions indexed by a set of parameters $θ$
- State your beliefs about $θ$ using a prior probability distribution, $p(θ)$
- Collect the data
- Use Bayes’ theorem to calculate your new (posterior) beliefs about $θ$
- Criticize the model
When and why do we sample from the posterior to obtain moments such as the mean and variance?
available. In these cases, we can use computational methods to obtain moments of the posterior or a set of draws from it, e.g., importance sampling or Monte Carlo integration methods.
If we sample from the posterior, we can know it with arbitrary precision
