06 Normal Distribution: Theory of Estimation Flashcards
What does i.i.d. stand for?
independent and identically distributed
What is maximum likelihood estimation?
= method for estimating population parameter from sample data such that the probability (likelihood) of obtaining the observed data is maximized
Used when e.g. logarithmic transformation did not change the data enough to meet normality assumptions
What are θ and x in the likelihood function L(θ;x)?
- θ is the distribution parameter vector
- x is the set of observations.
Why do we use the log-likelihood function?
- Since we have terms in product, we need to apply the chain rule which is quite cumbersome with products
- clever trick would be to take log of the likelihood function and maximize the same. This will convert the product to sum and since log is a strictly increasing function, it would not impact the resulting value of θ
- SO ALL IN ALL: to make differentiation (=maximization problem) easier
How is (log-)likelihood estimation related to OLS?
when the model is assumed to be Gaussian, the MLE estimates are equivalent to the ordinary least squares method
How does θ look like in the maximum likelihood estimation?
it is a vector of all unknown parameters. thus it has as many elements as there are unknown parameters.
How can you find the maximum likelihood estimate ~θ when given the distribution, θ and d?
1) find the pdf f (xᵢ; θ)
2) make it a likelihood function and then a log-likelihood function
3) take the first derivative wrt θ
4) set equal to 0ₚ and solve for ~θ
What is the first derivative of (XᵀX) ? The second derivative?
(XᵀX)' = 2 Iₚ X (XᵀX)'' = 2 Iₚ
What is the first derivative of (θᵀθ) ? The second derivative?
(θᵀθ)' = 2 Iₚ θ (θᵀθ)'' = 2 Iₚ
When θ = {μ, Σ}, how do you go about finding the MLE?
1) find the pdf f (xᵢ; θ)
2) make it a likelihood function and then a log-likelihood function
3) take the first derivative wrt μ
4) set equal to 0ₚ and solve for ~μ
5) use the information for ~μ to find the formula for ~Σ
