Topic 3: The Geometry of Linear Regression Flashcards
What is the difference between the following?
- RN is a set of n-vectors.
- More operations (like scalar products and length) are defined on Euclidean space.
What is the meaning of this operation?
Called the scalar or inner product of the two vectors.
Note that this operation is communative.
What is the meaning of the following operation?
The length of vector x.
In euclidean space this is calculated by:
How is the crossproduct of two vectors related to the angle between them?
Note that cos θ is 1 when the vectors are parralel, and zero when they are at right angles.
What does <x,y> = 0 imply?
That if x & y are nonzero vectors, they are at right angles.
They are said to be orthogonal, or perpendicular.
What is the Cauchy-Schwartz inequality?
Shown as below, which follows from the relationship between cross products and length, and the knowledge that the cosine of an angle is between -1 & 1.
What is a subspace? How can it be defined?
Can be defined by a set of basis vectors.
The subspace associated with a set of basis vectors is denoted S(x1, … sk),
The basis vectors are said to span the subspace, which is k dimensional in general.
The subspace consists of every vector that can be formed as a linear combination of the basis vectors.
What is an orthogonal complement?
Denoted as below, a subspace defined by a set of basis vectors that are orthogonal to everything in another (given) subspace.
Give a diagram showing the geometry of a linear regression model, as taken straight from the population. i.e. Parameters, not estimates. Explain.
This diagram follows from the two dimensional case.
This is because:
yi = XBi + ui
Where yi is the ith row of column vector y, etc.
So for the x axis, the width of the y vector (the first observation of y) is equal to the width of XB plus the width of u. So to for the y axis. For every observation, another dimension is added.
Remember that XB1 need not be zero, (this is simply viewing the triangle from the perspective of XB).
What is the meaning of the coordinates of a point in the subspace (X), where X is the matrix of variables from a regression model.
The coordinates are the values of the different explanitory variables in one observation.
What subspace does the n-vector Xβ belong to?
The subspace of X, as postmultiplying by the column vector β is mearly picking a different linear combination of the columns of X.
What is the geometric intepretation of the method of moments condition below?
Each row of this equation is a scalar product.
From the rule for selecting a single row of a matrix product, we get the following.
Where xi is the ith column of X. (Such that when transposed we have selected the row.)
Use a diagram to explain the Geometric interpretation of the method of moments estimation, with three observations.
This means we have two regressors.
Note that figure a) has three dimensions for all three observations, and that though x1 and x2 lie flat on the viewing angle, y is certainly not the only vector with a non-zero third observation. (Changing the viewing angle of this diagram does not affect the relationship between the variables or the estimation process.)
What is the implication of the following diagram, that follows from the relationship between the length of the vectors?
Total sum of squares = Explained sum of squares + Sum of squared residuals.
TSS = ESS + SSR
From:
Explain descriptively the orthogonal projection.
Maps points onto the closest point of the target subspace.
What is the formula for an orthogonal projection?
Show that calculating estimates of y with the method of moments estimator is the same projecting y orthogonally onto X.
For any vectors w and z,
What is the result of <PXw , MXw>?
Zero.
