Linear Algebra Theory Flashcards
What is a vector - mathematical interpretation
A vector is an ordered list of numbers
Number of elements in a vector is called
Dimensionality
What is a vector - geometric interpretation
A vector is a straight line with some length and some direction.
Vector and coordinate position overlap at
Standard position
What are the rules for vector addition or subtraction
The vectors to be added or subtracted must have same dimensions
What scales a vector up and down?
Scalar
Scalar is denoted by
greek letters - lambda, beta, alpha
Scalar stretches a vector when
the scalar value is greater than 1
Scalar flips a vector when
the scalar value is less than 1
when scalar value is 0
The vector is at origin
When a scalar is multiplied to a vector - it doesn’t change two things __________ and isn’t that notion contrary to what we know about when the scalar is negative?
- Direction and angle of the vector.
- Usually it is assumed the vector lies on a 1-D subspace that stretches in infinite direction.
What are the several ways of performing vector-vector multiplication?
- Hadamard multiplication
- Dot product
- Cross product
- Outer product
What is dot product?
Dot product provides a single number that provides the relationship information between two vectors.
Dot product is also called as
Scalar product
Mathematical notations for dot product
alpha = <a,b> = aTb = summation i = 1 to n ai*bi
Rules to perform dot product on two vectors
Both the vectors must have same dimensions.
What are the properties to which dot product is true
- Commutative
- Distributive
What are the operations in which the dimensionality of the two vectors must be equal
- Vector addition
- Vector subtraction
- Vector dot product
- Hadamard multiplication
What is commutative property in terms dot product?
a.b = b.a
What is distributive property in terms of dot product
a.(b+c) = (a.b)+(a.c)
For which properties is the dot product not true?
Associative property
What is associative property in terms of dot product?
a.(b.c) != (a.b).c
Definition of vector norm
Square root of the sum of each component squared.
Formula for vector norm
sqrt(vT. V)
What are the other names for vector norm
Vector length or vector magnitude.
Dot product - geometric interpretation
alpha = |a|.|b|.cos(theta(a,b))
How to get the angle between two vectors
theta = acos((a.b) or alpha/|a|.|b|)
Cosine is bound by
-1 and 1
If theta < 90 degrees then alpha or dot product is
> 90 degrees
If theta > 90 then alpha or dot product is
< 90 degrees
If theta is 90 degrees then alpha is ____________ and that vector is also called as _____________
0; orthogonal vector
If theta is 0 degrees then alpha is
|a||b|
If theta is 180 degrees then alpha is
- |a||b|
What is cauchy schwarz inequality?
the absolute value of the dot product between two vectors is less than or equal to the product of magnitudes of the two vectors
|aTb| <= ||a||*||b||
Is the dot product sign affected by scalar multiplication
Yes
Hadamard multiplication is also called as
Element wise multiplication
What is the rule for Hadamard or element-wise multiplication
Both the vectors must be of same dimension
Formula for outer product
outer(v,w) = v.wT
What is the rule for outer product
The vectors don’t have to be of same dimension.
The output of outer product is
matrix
What is the rule for vector cross product
Vector cross product can only be performed on 2 3-Dimensional vectors.
What is the output of vector product - dimensionality
3D Vector
Formula for cross product
[1;2;3] cross [a;b;c] = [2c-3b; 3a-1c; 1b-2a]
What does cross product represent
If two vectors v1 and v2 are used to represent a plane then the output vector v3 represent a vector perpendicular to that plane. (do fact check)
Complex numbers have angle with respect to
Positive real axis
Complex number format
a + bi
i^2 =
-1
i in complex number value is
sqrt(-1)
Complex numbers have ________ or _________ away from the origin of the complex plane
distance or magnitude
Hermitian transpose is also called as
Conjugate transpose
Hermitian transpose is performed for ________________
Complex vectors and complex matrices
The complex conjugate of a+bi
a-bi
Dot product between two complex vectors - v,w
vH. w or v*.W
Dot product between two complex vectors - v,w
vH. w or v*.W
What is a unit vector
A vector of length 1 is called unit vector.
How to normalize a vector?
mu = 1/norm(v)
For normalization
mu.v = normalized vector
norm(normalized vector) = 1
The magnitude of the dot product is _____ value of the dot product
absolute value of the dot product
What is dimensions
Dimension is the number of elements
What are fields
Field is a set of numbers on which addition, subtraction, multiplication and division are valid operations.
Example of fields -
Real numbers, complex numbers
Example of non-fields
Integers/counting numbers.
What is a subspace
A subspace is defined as the set of all vectors that can be created by taking the linear combination of some vector or set of vectors.
Linear combination is
- The multiplication of a vector by a scalar
- Addition of two vectors.
The scalar in scalar-vector multiplication can be ______
any real valued number
What is the formal definition for subspace
A vector subspace must be closed under vector addition, scalar multiplication and must contain zero vector.
Different vector from _____________ planes
Different
All the planes formed from different vectors converge at
origin
Ambient 3D space contains
- 0D Subspace
- infinite 1D subspace
- Infinite 2D subspace
- One 3D subspace
Why does having 2 vectors doesn’t mean they form a plane
The formation of the plane depends on if one vector is dependent or independent on the other vector
What is a subset?
Subset is a set of some points that satisfies some conditions.
What are the conditions for a subset?
- Doesn’t need to include origin
- Doesn’t need to be closed
- Can have boundaries.
What is span?
A region of space that you can reach by the linear combination of given vectors. The vectors ‘span’ that subspace.
What is thumb rule for dependent vectors
Any set of M > N vectors in RN - dependent vectors.
What is the thumb rule for independent vectors
Any set of M <= N vectors in RN - ‘COULD BE’ independent vectors.
Rules for basis
- Mutually orthogonal - 90 degrees
- Basis must be independent vectors
- We can have different basis
Matrix terminology
- Diagonal elements
- Off diagonal elements
Rectangular matrices or
Non-square matrices
Matrix size - format
rows X columns
What is a symmetric matrix
Square Matrix elements are mirrored across the diagonal.
Square matrices
- Symmetric matrix,
- Skew symmetric matrix
- Identity matrix
What is a skew symmetric matrix
Square Matrix elements are mirrored across the diagonal with flipped signs - with 0’s as its diagonal.
What values fit for a skew-symmetrical matrix diagonal?
0’s
Identity matrix
A square matrix with 1’s as diagonal elements and 0’s as off-diagonal elements.
Identity matrix is also a
Symmetric matrix
Multiplication property of identity matrix
A * I = A
Zero matrix
A matrix with all diagonal and off diagonal elements as 0
Diagonal matrix
A square or non-square matrix with zero or non-zero diagonal elements and zero off-diagonal elements is called as diagonal matrix
Relationship between diagonal matrix and identity matrix
D.I = D
Types of triangular matrices
- Upper triangular matrix
- Lower triangular matrix
What is upper triangular matrix?
- Elements above the diagonal can be zero or non-zero
- Elements below the diagonal are all zeros.
What is Lower triangular matrix?
- Elements below the diagonal can be zero or non-zero.
- Elements above the diagonal must be all zeros
What is the rule for concatenating matrices
The two concatenating matrices must have same number of rows.
Rule for matrix addition/subtraction
Must have same dimensions
Matrix addition/subtraction properties
- Commutative - A+B = B+A
- Associative - (A+B)+C = A+(B+C)
What is shifting a matrix?
Adding a scaled version of identity matrix
Properties of matrix shifting
- Doesn’t change off-diagonal elements.
- Changes diagonal elements
- Inflates a matrix
What is a linear operation?
Closed under matrix addition and scalar-matrix multiplication.
Is matrix-scalar multiplication a linear operation?
Yes
1. l(A+B)= lA + lB
2. la - al??
Transpose-Transpose of a matrix A
A.T.T = A
Transpose op vs symmetric matrix
For a symmetric matrix A - A.T = A
Transpose op vs skew-symmetric matrix
For a skew symmetric matrix A- A.T = -A
Hermitian transpose
Swaps signs for complex numbers
Matrix diagonal operation
Extracts the diagonal elements into a vector
Trace of a matrix
Sum of the diagonal elements of a matrix
Matrix types that diagonal and trace operations work on
- Diagonal operation - square and non-square matrix
- Trace operation- Only square matrices
Is trace a linear operator
Yes
1. trace(A+B) = tr(A) + tr(B)
2. trace(l * A) = l * trace(A)
What is broadcasting matrix arithmetic?
Addition of matrix and vector is invalid in traditional linear algebra. But via broadcasting the operation can be performed in modern machines. The row or column vectors are expanded as per the size of the matrix and then the operation is performed.
Matrix multiplication is valid when -
MXN * NM = MM matrix
5x7 * 5x2 matrix multiplication is
invalid
Why vector vector dot product is scalar
v1, v2 = 5 x 1
v1.Tv2 = 1x5 * 51 = 1x1
Ways to think about matrix multiplication
- Ordered collection of dot products
- Layered perspective
- Column perspective
- Row perspective
What is ordered collection of dot products perspective?
- Similar to vector-vector dot product
(pin on right column, iterate on left rows - For each iteration/row analysis - element-wise and add both) - First row * first column - element-wise
[0 1] * a
c
= 0*a + 1.c - Second row * first column- element-wise
* a
[2 3] c
= 2 * a + 3.c - First row * third column
- Second row * third column
What is layered perspective
1 left column to 1 right row - no element wise addition - forms a matrix
2nd left column to 2nd right row
and add both
- First column * first row - no element wise addition
0 * [a b]
2
= 0a 0b
2a 2b
- Second column * second row
1 * [c d]
3
= 1c d
3c 3d
0a 0b. +. 1c 1d
2a. 2b. 3c 3d
addition
1c 1d
2a+3c. 2b+3d
What is column perspective
- Linear weighted combinations of the columns on left matrix
(pin on right column, for each element in right column, multiply it with respective right column)
first element of the left column * first right column +
second element of the left column * second right column - First left column * first element of the first column (scalar vector multiplication) +
second left column * second element of the first column - first left column * first element of the first column + second left column * second element of second column
What is row perspective
first element of the left row * first right row +
second element of the right row * second right row
Order of operation
(EVIL).T = L.T * I.T * V.T * E.T
Matrix vector multiplication - op is
vector
Matrix vector multiplication - vector decides ______
orientation of output vector
Matrix vector multiplication - matrix decides ____________
the size of the output vector
For a rotation matrix
- No stretching
- Vector maintains its length
- Vector rotates by some angle
What is pure rotation matrix
Only rotation, no stretching
Impure rotation matrix
Rotation and stretching
What is Eigen vector and Eigen value
2 1. *. 1
2 3. 2
4
= 8
This can be expressed as 4 * 1
2
There is nothing special about the matrix or vector, but there is something unique about the combination of the matrix and vector.
When you can express the matrix-vector multiplication as some scalar-vector multiplication - the vector 1
2
is called Eigen vector and the scalar 4 is called the eigen value of the matrix.
What is the fundamental eigen value equation
AV = scalar.v
What are the two matrix identities
- Additive identity - A + 0 matrix = A
- Multiplicative identity - A * I = A
Create symmetric matrix from square non-symmetric matrix
S = (A+A.T)
or
S = (A + A.T)/2
Create symmetric matrix from non-square non-symmetric matrix
S = A * A.T
S = A.T * A
Sum of two symmetric matrices is
Symmetric matrix
Hadamard multiplication of symmetric matrices is
Symmetric matrix
Matrix multiplication of symmetric matrices
Non-symmetric matrix
How to get symmetric matrix from the multiplication of two symmetric matrices?
By adding constraints - a=d; e=g
a b. e f
c d f g
Does adding constraints for getting symmetric matrix from the multiplication of two symmetric matrices work for all matrix sizes?
No. only works for 2X2. Test - doesn’t work for 10X10
For diagonal matrix D, what is the result of DD, D.D (multiplication and hadamard multiplication respectively)
They both result in same result
How to compute Frobenius dot product via vectorization?
- Vectorize the matrices => vectors
- Compute the dot product of the vectors.
How to compute Frobenius dot product via transpose-trace?
- trace((A.T).B)
What is the best computational way to compute the Frobenius product?
Via transpose-trace matrix
What is Frobenius norm
Frobenius for product with itself is called Frobenius norm
Frobenius norm is also called as
Euclidean norm
Frobenius norm formula
norm(A) = √<A,A>F = √tr((A.T).A)
What is P-norm formula
||A||p = sup ||Ax||p/||x||p;
x != 0
What is p-norm
- P-norm gives the effect of matrix A on some vector x
- We care if Ax is longer or shorter than x
What is 2-norm
If we replace p with 2 in p-norm, it called 2-norm
When is ||Ax||/||x|| = 1
If A is a pure-rotation matrix or orthogonal matrix, it rotates x but keeps the length of the vector same.
Schatten P-norm definition
Sum of all singular values of a matrix.
What are singular values?
- Set of numbers
- It is a set of scalars thats associated with every matrix
How to identify singular values?
Goal of SVD is to identify the singular values.
Schatten P-norm formula
||A||p = ( (summation i to r) sigma i to the power p) whole power 1/p
If p = 1,
Ignore 1/p and called schatten 1-norm
What is a self-adjoint matrix?
A linear transformation that is represented by a matrix is said to be self-adjoint or self-adjoint operator if the following statement is true
<Av, w> = <v, Aw>
Rules for self-adjoint matrix
- A is square matrix
- A is symmetric matrix
- v and w are of same size (m X 1)
What is matrix asymmetry index?
Computes a value/index on how symmetric a matrix is
Matrix asymmetry index formula
ai = ||A~|| / ||A|| => Ratio of norms
||A|| = √tr(A.A) = √tr((A.T).A)
A~ = (A-A.T)/2 (Asymmetric part of A)
Matrix asymmetry index for symmetry matrix
0 since (A-A.T) = 0
Matrix asymmetry index for asymmetric matrix
With A-A.t we are actually doubling the matrix and then dividing by 2. This means ||A~|| == ||A|| = > the index is 1
Matrix symmetry index
1 - (||A~||/||A||)
