Prelim Exam Prep Deck

Question 1

Sandwich (robust) estimator for variance is given by ….

Answer 1

Question 2

Gibbs sampler is given by

Answer 2

Question 3

What’s consistency?

Answer 3

Question 4

What’s the Fisher information matrix?

Answer 4

Question 5

M-estimator

Answer 5

Any solution to \sum \psi(Y_i, theta) = 0

\psi does not depend on i or n.

true parameter value of theta is given by E\psi(Y_i, theta) =0

distribution (derived through Taylor expansion) is of form

\hat theta \sim AN (theta_0, V(theta_0) / n )

V= A^-1 B A^(-1T)

See sandwich variance estimator for more details

Question 6

What’s the gamma-exponential model?

Answer 6

prior p(theta) = Gamma(alpha, beta)

&

likelihood p( y | theta) = Exponential( theta )

> >

posterior p(theta | y) = Gamma(alpha + 1, beta + y)

Question 7

What’s the Poisson-Gamma Model?

Answer 7

Question 8

What are Jeffrey’s priors?

Answer 8

Question 9

What’s the normal-normal model?

Answer 9

Question 10

What’s the beta-binomial model?

Answer 10

Question 11

What’s the Metropolis Algorithm?

Answer 11

Question 12

What’s the Metropolis-Hastings algorithm?

Answer 12

Question 13

What’s importance sampling?

Answer 13

Question 14

What’s the normal pdf?

Answer 14

Question 15

What’s the gamma pdf?

Answer 15

Question 16

What’s the gamma mean?

Answer 16

Question 17

What’s the weak law of large numbers?

Answer 17

Question 18

What’s the CLT?

Answer 18

Question 19

What’s asymptotic normality of the MLE?

Answer 19

Question 20

What’s the Gauss-Markov Theorem?

Answer 20

The Gauss Markov theorem says that, under certain conditions, the ordinary least squares (OLS) estimator of the coefficients of a linear regression model is the best linear unbiased estimator (BLUE), that is, the estimator that has the smallest variance among those that are unbiased and linear in the observed output variables.

Question 21

What are the conditions required for the Gauss-Markov Theorem to apply?

Answer 21

independent/ uncorrelated error is the key.

Question 22

What’s the OLS estimator?

Answer 22

Question 23

What’s the sum of an infinite power series?

Answer 23

Question 24

What’s the sum of an infinite geometric series?

Answer 24

Question 25

What’s Small o: convergence in probability?

Answer 25

Question 26

What’s Big O: stochastic boundedness?

Answer 26

Question 27

What’s the EM algorithm?

Answer 27

Question 28

What’s an orthogonal projection?

Answer 28

The orthogonal projection of a vector $s$ onto a given subspace $R$ is the vector $rin R$ that is closest to $s$.

Question 29

What does it mean for a projection to be idempotent?

Answer 29

Question 30

What’s Newton’s Method?

Answer 30

Question 31

When/ how might Monte Carlo Integration be useful for likelihood estimation?

Answer 31

Question 32

What are M-estimators and how are they different from MLE?

Answer 32

Question 33

What’s the Poisson PMF?

Answer 33

Question 34

When might consistency/ asymptotic approximation of the MLE break down?

Answer 34

If the number of parameters increases with the sample size, the usual theorem about consistency/ asymptotic approximation (normality) of MLEs doesn’t apply.

Question 35

When might we use the delta method?

Answer 35

The Delta method is a theorem that can be used to derive the distribution of a function of an asymptotically normal variable.

It is often used to derive standard errors and confidence intervals for functions of parameters whose estimators are asymptotically normal.

It can be useful in contexts where asymptotic properties of the MLE breakdown e.g. the number of parameters increases with sample size and the fisher’s information consequently doesn’t approximate the asymptotic variance.

Question 36

What are the assumptions for OLS regression?

Answer 36

1. Linearity in parameters (variable transformation can be used to meet this assumption but may make interpretation difficult)
2. Full column rank X (When a matrix is rank deficient (not full rank), its determinant becomes zero and subsequently, its inverse does not exist)
   - If perfect multicollinearity exists between any two or more regressors, matrix X does not have “k” linearly independent columns. It is, therefore, not full rank.
   - If n<k then the rank of the matrix is less than k, it does not have full column rank. Therefore, when we have k regressors, we should have at least k observations for being able to estimate β by OLS.
   - For simple linear regression, if all the values of the independent variables x are the same then we have two columns in matrix X — one having all 1’s (coefficient for the intercept term) and the other having all c (the constant value of the regressor), as shown below. It is evident that in this case too, the matrix is not full rank, the inverse does not exist, and β cannot be determined.
3. Zero Conditional Mean of Errors

Note: Conditions 1-3 mean the OLS is unbiased.

To have the OLS as the BLUE by Gauss Markov Theorem:

4. Homoscedasticity: we do not want the regressors to carry any useful information regarding the errors.
5. Nonautocorrelation: Errors are unrelated to other errors.

Additional assumption:
6. Normality
- When all the other assumptions are met along with the normality assumption, OLS estimates coincide with the Maximum Likelihood estimates, giving us certain useful properties to work with.
- Coefficient estimates for βᵢ can be shown to be a linear function of errors ϵᵢ. A very cool property to keep in mind— a linear function of a normally distributed random variable is also normally distributed. Therefore assuming ϵᵢ as normally distributed gives us βᵢ estimates as normally distributed as well. * This makes it easier for us to calculate confidence intervals and calculate p-values for the estimated β — coefficients (which are commonly seen in R and Python model summaries). If the error normality condition is not satisfied, then all the confidence intervals and p-values of individual t-tests for β — coefficients are unreliable.

Source: https://towardsdatascience.com/assumptions-in-ols-regression-why-do-they-matter-9501c800787d#:~:text=What%20does%20it%20mean%20for,linearly%20independent%20of%20each%20other.

Question 37

What is Iteratively reweighted least squares?

Answer 37

The method of iteratively reweighted least squares (IRLS) is used to solve certain optimization problems with objective functions of the form of a p-norm, and can be used to estimate the beta coefficients for a generalized linear model.

Question 38

What’s the invariance property of the MLE?

Answer 38

Question 39

What’s Bayes Rule?

Answer 39

Question 40

What’s the definition of correlation?

Answer 40

Question 41

What’s standard brownian motion?

Answer 41

Question 42

What’s the variance of the sum of two random variables?

Answer 42

Question 43

What’s a Poisson process?

Answer 43

Question 44

What’s the exponential distribution?

Answer 44

Note, the exponential distribution is memoryless.

Question 45

What’s the connection between the exponential distribution and the gamma?

Answer 45

Sum of exponentials(lambda) = gamma(n, lambda)

Question 46

What’s the general form for the f-statistic?

Answer 46

Question 47

What’s the Kronecker product of A (an m × n matrix) and B is (a p × q matrix)?

Answer 47

Question 48

What’s the bernoulli pdf?

Answer 48

EK=p and VarK=p(1-p)

I think the pdf might actually be
f = p^X(1-p)^(1-X)

Question 49

What’s the binomial pdf?

Answer 49

EX=np and VarX=np(1-p)

Question 50

What’s variance in terms of expectation?

Answer 50

Note what variance reduces to when E(X) = 0.

Question 51

What’s the law of total expectation?

Answer 51

Question 52

What is the E( beta(a,b) )?

Answer 52

Question 53

What’s a useful gamma function identity?

Answer 53

Question 54

What’s the Bayes Factor for model comparison?

Answer 54

Question 55

What’s the beta-binomial model’s marginal?

Answer 55

Question 56

Answer 56

Question 57

Answer 57

Question 58

a) time to state change?

Answer 58

Question 59

b) generator matrix?

Answer 59

Question 60

c) how might we use the generator matrix to get the stationary probabilities?

Answer 60

Question 61

What’s the difference between Bayesian and frequentist residuals in the context of Linear Models?

Answer 61

Question 62

What’s the equivalent Bayesian setup of a frequentist linear regression?

Answer 62

Question 63

What’s the equivalent Bayesian setup of a frequentist penalized regression?

Answer 63

Question 64

What’s the beta function in terms of the gamma function?

Answer 64