3) Maximum Likelihood Estimation

Question 1

What is the Likelihood Function and Log-Likelihood Function

Answer 1

Question 2

What is the score function

Answer 2

The first derivative of the log-likelihood function with regard to the parameters

Question 3

What steps are involved in finding the Maximum Likelihood Estimate (MLE)

Answer 3

• The condition for MLE is 𝑺𝑛(𝜽ˆ 𝑀𝐿) = 0 indicating the point where the log-likelihood function’s slope is zero, suggesting a maximum or minimum.
• Confirming a Maximum:
• Scalar θ: Ensure the second derivative of the log-likelihood is negative confirming maximum at 𝜽ˆ𝑀𝐿
• Vector θ: Compute the Hessian matrix at the MLE. The MLE is confirmed if this matrix is negative definite (all eigenvalues are negative), ensuring the log-likelihood is locally concave.

Question 4

How can cross-entropy and KL divergence be approximated for large sample sizes

Answer 4

For large sample size n, the cross-entropy 𝐻(𝐹, 𝐺𝜽) is approximated by the empirical cross-entropy where the expectation is taken with regard to 𝐹ˆ𝑛 rather than 𝐹

Question 5

How is maximizing the log-likelihood related to minimizing KL divergence for large sample sizes

Answer 5

Question 6

What does it mean for an estimator to be consistent

Answer 6

Question 7

Does the maximum likelihood function produce consistent estimates

Answer 7

Yes the method of maximum likelihood in general produces consistent estimates

Question 8

Why are MLEs considered asymptotically unbiased

Answer 8

Maximum Likelihood Estimates (MLEs) are considered asymptotically unbiased because, with sufficient data, the MLE will converge to the true parameter value

Question 9

How does maximum likelihood estimation connect to least squares estimation in a normal distribution model

Answer 9

In a normal distribution model, finding the mean using maximum likelihood estimation is equivalent to using least squares estimation.

Question 10

What is the observed Fisher information

Answer 10

The negative curvature at the MLE:
𝑱𝑛(𝜽ˆ𝑀𝐿), the derivative of the negative score function evaluated at the MLE

Question 11

What is the relationship between observed and expected Fisher information

Answer 11