lecture 1 - matrix algebra Flashcards
properties of matrices
- inverse property
- transpose of product
- inverse of product
- transpose of inverse
- double transpose
- distributive property for transpose
What is a Symmetric Matrix?
- satisfies A^⊤ = A
- means the matrix is equal to its transpose
- Elements across the main diagonal are mirrored.
- mathematically: a_{ij} = a_{ji}
Definition of Positive Definite Matrix
a matrix A is positive definite if x^⊤Ax > 0 for all non-zero vectors x, given that A is symmetric
Why are Positive Definite Matrices Important in Machine Learning?
- Indicates strictly convex functions, leading to optimal solutions.
- In machine learning, this guarantees that algorithms like linear regression converge to a single, optimal set of parameters.
What is the result of multiplying a matrix and a vector?
- When you multiply a matrix A and a column vector x, the result is a new column vector.
- Each element of the new vector corresponds to a weighted sum of the matrix’s rows and the vector’s elements.
How do you compute the elements of the result when multiplying a matrix by a vector? (Ax_i)
To find the value of a specific row in the resulting vector:
- Multiply each element in that row of the matrix by the corresponding element in the column vector.
- Add these products together to get the final value.
- (Ax)i = \sum^n{j=1} a_{ij}x_j
What does each element in the resulting vector Ax represent?
- Each element represents the “weighted sum” of one row of the matrix, weighted by the values of the column vector.
- row of matrix * vector column
What does differentiating a scalar with respect to a vector produce?
- Produces another vector.
- The resulting vector contains partial derivatives of the scalar with respect to each element of the original vector.
How do you differentiate a scalar-vector product with respect to a vector?
- e.g., x^T a
- only the terns involving the current varianble x_i survive
- the result is a transposed vector (i.e., a row vector)
Why do we transpose the result of vector differentiation?
The transpose is necessary because gradients are conventionally row vectors. This ensures that when multiplied by another vector, the matrix dimensions align correctly.
What happens when you differentiate b^T Ax with respect to x?
- A^T b
- This result comes from applying the scalar-vector differentiation rule, where only terms involving the current variable x_i survive, and the coefficients form the product A^T b
What does differentiating a vector with respect to a vector produce?
A jacobian matrix
What is the Jacobian matrix?
- The Jacobian matrix is the derivative of a vector-valued function with respect to a vector of variables.
- It describes how each component of the function changes with respect to each variable.
What does the Hessian matrix represent?
- The Hessian matrix is a square matrix of second-order partial derivatives of a scalar-valued function.
- It provides information about the curvature of the function.
How does the Hessian matrix help determine the nature of a critical point?
- Positive eigenvalues: The function is convex, indicating a local minimum.
- Negative eigenvalues: The function is concave, indicating a local maximum.
- Mixed eigenvalues: The function has a saddle point, meaning it curves up in one direction and down in another.
