Maximum likelihood estimation

Question 1

Log-likelihood for GLMs

Answer 1

θ = (β, σ²)
l(θ) = -n/2 ln2π - n/2lnσ² -1/(2σ²) (y-Xβ)’(y-Xβ)

Question 2

Maximum likelihood estimator for β

Answer 2

β^ = argmax(β ∈ R^p) l(β, σ²) = (X’X)^-1X’y
Equivalent to OLS estimator

Question 3

Properties of β^

Answer 3

• β^|X ~ N_p(β, σ^2(X’X)^-1));
• Unbiased;
• Efficient in the Cramer-Rao sense.

Question 4

Maximum likelihood estimator for σ²

Answer 4

σ²^_ML = argmax(σ² ∈ R) l(β, σ²) = ε^’ε^ / n = SSE / n
Raw residuals ε^ = y - y^ = y - X(X’X)^-1X’y = [In - X(X’X)^-1X’] y = (In - H)y

Question 5

Unbiased estimator for σ²

Answer 5

σ²^= SSE / (n-p)
Also obtainable thorugh the restricted liklelihood function maximisation. It is unbiased and independent of β^ too.

Question 6

Properties related to σ²^_ML

Answer 6

• ε^|X ~Nn(0, σ²(In - H) that can be standardized into r_i = ε^_i / √ σ²(1-Hii) and then r|X -d-> N_n(0, In)
• SSE/σ² ~ χ²_n-p, which leads to E[σ²_ML] = σ² (n-p)/n so biased even in asimptotically unbiased
• SSE/σ² ⊥ β^ => σ²_ML ⊥ β^