Chapter 9: Conjugate Priors

Question 1

Suppose that we run a restaurant and want to build a model for the number of people who have food allergies in a particular sitting, X, in order to inform the buying of ingredients. If we assume that the allergy status of one person is independent of everyone else’s what is a reasonable model choice?

Answer 1

Binomial sampling model X ∼ B(n,θ ), where n is the number of people in a sitting and θ is the probability that a randomly chosen individual has an allergy.

We can write the binomial likelihood as:
P r ( X = k | θ , n ) = (|n,k|) θ^k ( 1 − θ )^(n − k)
∝ θ^k ( 1 − θ )^(n − k)

Question 2

What would therefore be a good prior for θ and why?

Answer 2

Since the beta distribution is defined over the [0,1] interval and can represent a range of prior beliefs, we use a beta prior for θ

Question 3

How is the beta distributions probability density function (pdf) written as?

Answer 3

p(θ | α , β ) = θ^a-1 (1 - θ)^B-1 / B(a,b)
∝ θ^α−1 (1−θ)^β−1

where B(α,β) is a beta function, which is not dependent on θ.

Question 4

What about the sampling distribution:
θ^k ( 1 − θ )^(n − k)
and the prior distribution:
θ^α−1 (1−θ)^β−1
stand out and how is this useful?

Answer 4

We notice that the expression for the sampling distribution and the expression for the prior both contain a term of the form θa(1−θ)b. When we use Bayes’ rule to calculate the posterior, we are required (in the numerator) to multiply together the likelihood and the prior, resulting in:

p(θ |data)∝ p(data|θ)× p(θ)
∝θ^k (1−θ)^(n−k) ×θ^(α−1) (1−θ)^β−1
=θ^(k+α−1) (1−θ)^(n−k+β−1)
=θ^(α′−1) (1−θ)^(β′−1)

Question 5

What do α ′ and β ′ equal in this case (beta posterior)

Answer 5

α ′ = α + k and β ′ = β + n − k.

Question 6

If we actually do the denominator calculation in Bayes’ rule what do we find it equals in this case?

Answer 6

B(α′,β′), meaning that the posterior PDF is a beta distribution, although with different parameters to the prior.

Question 7

Here, what do we say the beta prior is relative to the binomial likelihood and why?

Answer 7

Here we say that the beta prior is conjugate to the binomial likelihood since the posterior is also a beta distribution.

Question 8

The following represents the bayesian inference process:

prior -{likelihood}→ posterior

How is this described in the former example?

Answer 8

beta -{binomial}→ beta′

Question 9

Why is beta’ used instaed of just beta?

Answer 9

Because it is a beta distribution but with updated parameters from the initial distribution

Question 10

Generalise this rule in regarding the relationship between prior and posterior distributions

Answer 10

Conjugate priors are always defined relative to a particular likelihood and means that both the prior and the posterior come from the same family of distributions. Diagrammatically, we have that for a specified likelihood, L, and a prior distribution from a particular family, f ∈ F:
f -{L}→ f ′

Question 11

If we feed a gamma prior into our Bayesian updating rule, we should get a gamma posterior out. But what would the likelihood be?

Answer 11

Poisson likelihood

Question 12

If we feed a normal prior in, we should get a normal posterior out. BUt what would the likelihood be?

Answer 12

for particular normal likelihoods

Question 13

Are there disadvantages to using conjugate priors?

Answer 13

While the use of conjugate priors makes Bayesian statistics easy, it can limit us. These limits are quickly approached when we need greater modelling flexibility, which is especially the case when using hierarchical models e.g if we notice that our data is more suited to a student’s t distribution than a normal distribution