Linear Models Flashcards
Recall the standard linear regression model, define any terms and indices
yi=B0+B1xi1+B2xi2+…+Bkxik+ei
yi is the ith response
xij is the ith value of the jth regressor
k is the number of regressors (k+1=p parameters)
B0 is the y-intercept
Bj is coefficient associated with the jth regressor
ei is an error term, usually assumed iid random with mean 0 and variance sig^2
Matrix form of regression model and what are any assumptions made (OLS assumptions)?
y = XB + e
E[e]=0
var(e)=sig2 I
OLS estimate of B variance and expected value
Bhat = (X’X)-1X’Y
Var(Bhat)= sig2(X’X)-1
E(Bhat)=B
What does OLS estimate, Bhat, do?
Minimizes the sum of squared residuals (in matrix form: SSres= [y-yhat]’ [y-yhat])
SSres in quadratic form (Y’AY where A is symmetric)
Y’[I - X(X’X)-1X’]Y
Global F-test
H0: B1=B2=…=Bk=0
Ha: Bj ≠ 0
Test statistic is MSreg / MSres ~ F
Where MSreg = SSreg / k
SSreg = SUM (yhati - ybar)2
Write SSreg in terms of matrix notation
SSreg = Y’ [X(X’X)-1X’ - 1(1’1)-11’]Y
Describe a cell means model
yij= ui + eij
yij is the jth observation for the ith group
i=1,2,…,t
j=1,2,…,ni
ui is the true mean for the ith group
eij is the error term
For a cell means model, define y, M, and u
What is the incidence matrix and what is uhat, M’M, (M’M)-1?
Less than Full Rank ANOVA Model (overly paramterized)
Where the row length of X is larger than the column length
Difference between overly parameterized and fully parameterized
Overly paramterized models have columns in the X matrix which are linear combinations of the first column (ie not orthogonal)
For less than full rank anova model, what happens to Bhat?
Since Bhat= (X’X)-1X’Y it cannot be estimated because X’X does not have an inverse. But, X’X does have generalized inverses which have to be solved that way
Consequences of Less than Full rank ANOVA models
Generalized inverses are not unique
There are no unique estimates for B
Serious limitations on what we can estimate and test
Usual focus: contrasts (pairwise comparisons, factorial effects)
Things of critical importance: eigenvalues and rank of matrices (provides degrees of freedom)
If A and B are nxn square matrices then det[AB]=
det[A]det[B}
If A is nxn then det[A]=0 iff
A is singular
The rank of A is…
the greatest number of linearly independent columns (or rows) of A. (Linear dependence implies that at least one column (or row) of A can be written as a linear combination of the other columns (or rows) )
If A and B are non-singular, then for any matix C
C, AC, CB, ACB
all have the same rank
If A is an mxn matrix of rank r, then there exist non-singular matrices P and Q such that PAQ is one of the following:
The rank of AB cannot
exceed the rank of either A or B
If A is a nxn matrix then det[A]=0 iff
the rank of A is less than n
The matrix of a quadratic form can always
be chosen to be symmetric
A and B are said to be congruent matrices iff
there exists a non-singular matrix, C, such that
B=C’AC
We say C is the congruent transformation of A
Let A be an nxn symmetric matrix of rank r. There exists a non-singular matrix C such that
C’AC=D
where D is a diagonal matrix with exactly r non-sero diagonal elements
If A and B are congruent matrices, then
they have the same rank
Let C be an mxn matrix with rank r then the ranks of C’C and CC’
are also r
Let A be an nxn matrix. There will always exist
n eigenvalues that satisfy
det[A-LI]=0
where L is a diagonal matrix of the n eigenvalues
Let A be an nxn symmetric matrix. The rank of A
is the number of non-sero eigenvalues
Let A be an nxn matrix. A has at least one zero eigenvalue iff
A is singular
Let A be an nxn matrix. The determinant of A is the
product of eigenvalues
Let A be an nxn matrix and let C be any nxn non-singular matrix then
A, C-1AC, CAC-1
all have the same number of eigenvalues
Let A be an nxn real matrix. A necessary and sufficient condition that there exist a nonzero y that satisfies Ay=ey
is that e be an eigenvalue of A
Let P be an nxn matrix. P is called orthonormal iff
P-1=P’ and therefore PP’=P’P=I
Let A be an nxn matrix, and let P be an nxn orthonormal matrix, then
det[A]=det[P’AP]
Let x and y be nx1 vectors. x and y are called orthogonal if
x’y=0 or y’x=0
Let A be a mxn matrix and B be an nxp matrix. A and B are said to be orthogonal if
AB=0
Let A be nxn symmetric matrix. There exists an orthonormal matrix P such that
P’AP=D where D is a diagonal matrix whose diagonal elements are the eigenvalues of A
Let A be an mxn matrix with rank r>0. There exist matrices AL (mxr with rank r) and AR (rxn with rank r) such that
A=ALAR
Define column and row space
Column space of a matrix A is the set of vectors that can be generated as linear combinations of the columns of A.
Row space of a matrix A is the set of vectors that can be generated as linear combinations of the rows of A.
Let A be an nxn matrix. A is called positive semidefinite if
a) A=A’ (A is symmetric)
b) y’Ay ≥ 0 for all y
c) There exists at least one y ≠ 0 such that y’Ay=0
If A, an nxn matrix, is positive semidefinite then (3 things)
1) The rank of A is less than n
2) The eigenvalues of A are greater than or equal to 0
3) If P is an nxn non-singular matrix then P’AP is also positive semidefinite
Let A be an nxn matrix. A is positive definite if
a) A=A’ (A is symmetric)
b) y’Ay > 0 for all y≠0
If A is positive definite then (3 things)
1) The rank of A is n
2) All of the eigenvalues of A are greater than 0
3) Let P be an nxn non-singular matrix. P’AP is also positive definite
A matrix is called non-negative definite if it is
either positive definite or positive semidefinite
Let C be an mxn matrix with rank r. C’C and CC’ are
both non-negative definite.
C’C or CC’ are positive definite iff they have full rank
Let A be an nxn symmetric non-negative definite matrix. There exists some nxn matrix B such that
B’B=A
Let A and B be nxn symmetric matrices. If A is positive definite, then there exists a non-singular matrix Q such that Q’AQ= ____ and Q’BQ=____
Q’AQ=I and Q’BQ=D where D is a diagonal matrix whose diagonal elements are the roots of det[B-lambdaA]
If A and B are both non-negative definite, then there exists a matrix Q such that both Q’AQ and Q’BQ are
diagonal
Let A be a nxn non-singular matrix partitioned into
A=[A11 A12
A21 A22]
where both A11 and A22 are square and non-singular and let A-1=C
What is C?
Let A be an nxnx matrix. We call A idempotent if
AA=A
If A is an nxn idempotent matrix with rank n, then A=
I
If A is an nxn idempotent matrix of rank less than n, then A is
positive semidefinite
If A is an idempotent matrix with rank r, then it has
r non-zero eigenvalues, each equal to 1
Let A be an nxn (symmetric) idempotent matrix then
a) A’ is
b) Let P be an orthonormal matrix. P’AP is
c) Let P be an nxn non-singular matrix. PAP-1 is
d) I-A is
a) (symmetric) idempotent matrix
b) (symmetric) idempotent matrix.
c) idempotent
d) (symmetric) idempotent matrix
The trace of an nxn matrix A is the
sum of the diagonal elements
Let A be an mxn matrix and let B be an nxm matrix.
trace[AB]=
trace[BA]
If A, B, C are conformable, then
trace[ABC]=
trace[CAB] = trace[BAC] = trace[BCA] = trace[CBA] = trace[ACB]
Let A be an nxn matrix and let P be a non-singular nxn matrix.
trace[A] =
if P is orthonormal then
trace[A]
trace[P-1AP]
trace[P’AP]
Let A be an nxn matrix with eigenvalues lam1, lam2, lam3, … , lamn
trace[A] =
SUM lami
Let A and B be nxn matrices and let a and b be scalars.
trace[aA+bB] =
a trace[A] + b trace[B]
Let A be an nxn matrix what does this say about the trace?
trace[A’] = trace[A]
Show the general form to obtain a symmetric matrix
A Moore-Penrose Inverse A+ for an mxn matrix A satisfies the following 4 conditions
1) AA+ is symmetric
2) A+A is symmetric
3) AA+A=A
4) A+AA+=A+
Let X be an nxp matrix with rank p. Then X+=
(X’X)-1X’
Let A be an mxn matrix. Then every matrix A has a Moore-Penrose inverse, and ______
it is unique
(A’)+=
(A+)’
The Moore-Penrose inverse of A+ is
A
If the rank of A is r, then each of the following matrices also has rank r (in terms of Moore-Penroses)
A+
AA+
A+A
If A is non-singular, then A+=
A-1
If A is symmetric idempotent, then A+=
A
The matrices AA+, A+A, I-AA+, I-A+A are all
symmetric idempotent
For any matrix A, (A’A)+=
A+(A’)+
Let P be an mxm orthonormal matrix. Let Q be an nxn orthonormal matrix, and let A be any mxn matrix. Then (PAQ)+=
Q’A+P’
Let X=[X1 X2]
Then XX+X1=
X1
Let A be an mxn matrix. A- is called a generalized inverse of A if
AA-A=A
Note, the Moore-Penrose inverse is also a generalized inverse
Let X be an mxn matrix with rank r > 0. The following conditions hold for the generalized inverse
What can be said about X(X’X)-X’
It is invariant to the choice of generalized inverse
X(X’X)-X’=
XX+
The following conditions hold for K=X(X’X)-X’
If A is an nxn non-singular matrix, then the system Ax=y
has a unique solution
The system Ax=y has a solution iff (2 answers)
y is in the column space of A
AA-y=y
Let A be an mxn matrix. The system Ax=0 has a solution other than x=0 iff
the rank of A < n
Let a and x be nx1 vectors. What are the derivatives with respect to x of
a’x, x’a, and x’x
a, a, and 2x
Let A be an mxn matrix of constants and let x be an nx1 vector. Then the derivative of Ax with respect to x is
A
What is the derivative with respect to x of x’Ax?
2Ax
What is a stationary point, x0?
x0 is a solution to df(x)/dx = 0
Could be a min response, max response, or a saddle point
What is the Hessian matrix and what information does it provide?
It is a matrix of second derivatives. It describes the funciton in the neighborhood of the stationary point.
Conditions for Ordinary Least Squares estimation of B
How to decompose variance-covariance matrix (where it is known that the matrix is symmetric)
What does the generalized least squares estimate of B do?
What is the GLS estimate of B?
What is var(a’y) and var(Ay)?
What is E[y’Ay]?
What are possible approaches toward minimizing the variance matrix of Bhat? (sig2(X’X)-1)
1) min |sig2(X’X)-1| which minimizes the volume of the confidence ellipsoid of the estimated coefficients
2) min tr[sig2(X’X)-1] which would minimize the sum of the variances of the estimated coefficients
Both put an emphasis on eigenvalues
Steps to show that an estimator is the best, linear, unbiased estimator
Describe the density function for y, a px1 vector from the multivariate normal distribution
If z is a nx1 univariate standard normal random variable then what can be said about z’z?
It is distributed chi-squared with n degrees of freedom
What is the non-centrality parameter?
Three things to consider if trying to show y’Ay follows a chi-squared and how to define non-centrality parameter
What test statistic should be used if Σ is known?
What test statistic should be used if σ2 is unknown but V is known, where σ2V=Σ?
Chi-squared
F-distribution
The classic ANOVA table
Source df MS E[MS] F lambda
Xc‘1=
var(e)=
Σ=
Σ-1=
Define the cell means model
Global F-test for cell means model
ANOVA for cell means global F-test
When using a contrast C for multiple comparison testing, what does the rank of C control?
The type I error rate for the set of comparisons
Describe an identifiable parameter
Describe an estimable funciton
Let A be an mxn matrix. The system Ax=0 has a solution other than x=0 iff
the rank of A<n>
</n>
Results of this
All contrasts are _____
estimable functions
What are the differences between pairwise comparisons for the cell means model and the effects model?
For the generalized inverse case, test statistic given L’Bhat for an effects model
Where whatever is being tested is in the row space of X (L’=AX)
Reasons the effects model is not very valuable from an analysis perspective
Resquires constraintes to be identifiable
Constraints ultimately must be linear combinations of the predicted values
Analysis must focus on estimable functions (contrasts are always estimable functions)
However, the effects model does generalize very nicely
RBD Model and X matrix
X=[1 T M]
Sets of Contrasts for RBD (to test treatments and blocks)
Way to make orthogonality for X’X RBD
ANOVA table for RBD.
Is there an appropriate test for fixed block effects?
RCB model where interaction is important
AB interaction contrasts for RBD interaction model
Creating orthogonality for RBD interaction model
ANOVA for RBD interaction model
Interaction effect for RBD model and significance in relation to contrasts
If AB is significant, contrasts must focus on the comparing cell means
If AB is not significant, then contrasts would focus on the marginal means
Model for one way random effects and variance of Y
Main thing to test for random effects model
ANOVA for one way random effects model
For two way random effects model define distributions for A, B, AB
ANOVA two way random effects model
Distribution for SS of A, B, AB for mixed model
ANOVA mixed model
