9) Quadraric Forms Orthogonal And Symmetric Matrices

Question 1

INNER PRODUCT

Properties

Answer 1

The inner product on R^n is the map:
The dot product
R^n X R^n -> R, (x,y) -> x•y =(x,y) 
= SUM( x_i y_i) = x^T y
Sum from I=1 to n.

Properties: 1) symmetry: x•y = y•x

2) bilinearity: for all x,y,x’,y’ in R^n and for all s,t in R: (sx + tx’) •y = sx•y + tx’•y and x•(sy+ty’) = sx •y + tx •y’
3) non negativity: x•x is always bigger than or equal to 0

Question 2

Length of vector x

Answer 2

|x| = SQRT( x•x) as always bigger than or equal to 0

x=0 if and only if the length of x is 0
So if length is 0 it’s the 0 vector

If x is not the 0 vector then x/ |x| has length 1 (UNIT VECTOR)

Question 3

Orthogonality for vectors if x,y in R^n are orthogonal

Answer 3

x,y in R^n are orthogonal if x•y =0

E.g. If inner product =0
If x^T y =0

Question 4

Orthonormal

Answer 4

A set if vectors v_1,v_2,…,v_n in R^n is called orthonormal if:

v_i • v_j = kronecker delta function = { 1 if I=j, 0 if i≉j}
Eh same vectors dot to give 1 -> length is 1
All others dot gives 0 -> orthogonal

Orthogonal set all with length 1

Question 5

Theorem for mutually orthogonal vectors

Answer 5

ANY SET OF:
n mutually orthogonal
Non-zero vectors
v_1,...v_n in R^n
Form a BASIS FOR R^n.

As they are linearly independent: shown by dot of each to give scalars must be 0. I.e. Taking inner product gives kronecker delta function.

Question 6

Orthogonal Matrices definition

Answer 6

A nXn matrix is orthogonal if A subset of M_n(R) (Real square Matrices):

Ax•Ay = x•y for all x and y in R^n.

(Ax)^T Ay = x^T A^T Ay =x^Ty

Hence ANY ORTHOGONAL matrix A has an inverse A-1 = A^T

A ∈O(n) if and only if it’s columns, as a set of vectors form an orthonormal set

Question 7

Orthogonal Matrices properties

Answer 7

The identity matrix E is an orthogonal matrix.
•if A and B are orthogonal Matrices then so is AB

• if A is an orthogonal matrix so is A^T = A^-1
• if A is an orthogonal matrix, then |A| + +1 or -1

Converse not true: there are plenty of Matrices with det A plus or minus 1 that aren’t orthogonal but given that it is orthogonal we know the determinant is either 1 or -1

Question 8

Orthogonal Matrices

• As a set

Answer 8

The set of all orthogonal nXn Matrices is a group O(n) = {A subset M_n(R) | A^T A =E}

= {A in M_n(R) | A^-1 = A^T }

• A ∈O(n) if and only if it’s columns, as a set of vectors form an orthonormal set by seeing that elements in matrix A^TA are identity matrix elements relating to kronecker delta function

Question 9

Examples of orthogonal Matrices

Answer 9

Orthogonal Matrices det 1 or -1 but converse not true.

Eh rotation matrix = (cosx, -sinx, sino, cosx)
And permutation Matrices

Question 10

Symmetric Matrices

Answer 10

A matrix A⊂M_n(R) or a linear map A: R^n -> R^n, x-> Ax
Is called symmetric if

Ax•y = x•Ay for all x,y in R^n

A MATRIX A IS SYMMETRIC IF AND ONLY IF A^T = A

Shown as applying symmetry to inner product and equating

Question 11

Theorem for symmetric matrix and eigenvalues and eigenvectors

Answer 11

Let A⊂M_n(R) be a SYMMETRIC MATRIX. Then:

1) all eigenvalues of A are REAL numbers
2) for vectors u and v. Au = tu and Av =sv and s≉t then

u•v =0
I.e. For eigenvectors of A with DISTINCT eigenvalues then they are orthogonal with inner product 0

Question 12

Symmetric matrix and basis

Answer 12

If A is an nXn symmetric matrix then there is an ORTHONORMAL basis for R^n consisting of eigenvectors for A.

BASIS FOR R^n

If NORMALISED THEY DO!!!

As if for n distinct eigenvalues for each eigenvector corresponding length one can be found. Then we have that the set is orthonormal as every dot gives required kronecker delta function.
Since characteristic equation for A is polynomial degree n even eigenvectors form a basis for R^n when eigenvalues not all distinct.

Question 13

Corollary of symmetric matrices and diagonal

Answer 13

If A is a symmetric nXn matrix then there is an orthogonal matrix P in O(n) and nXn diagonal matrix D st

A = PDP^-1 = PDP^T

Of NORMALISED EIGENVECTORS

For coordinates on V the orthonormal basis reduces to dimensions

Question 14

Quadratic forms

Answer 14

A quadratic form on R^n is a function of the form

Q(x) = ΣΣq_ij x_i y_j sums from i=1, j=1 to ∞

Set of n^2 constants q_ij in R.

Since we have x_ij = x_ji this implies that b_ij ={ q_ij if I=j, or mean of q_ij and q_ji if i not equal to j)

That is in matrix form coefficients of x_ix_j and x_jx_i are equal.

Symmetrically quadratic form is Q(x) = ΣΣb_ij x_i y_j Where b_ij = b_ji

Defined as a matrix:

Q(x) = x^T B x = (x_1,…,x_n) B Col_vector( x_1,…,x_n)
B is symmetric matrix due to coefficients b_ij.

Question 15

Example of quadratic form

Answer 15

• Q(x) = x_1 ^2 + 4 x_1 x_2 -3x_2 ^2 is a quadratic form in R^2 with coefficients: q_11= 1, q_12= 4, q_21=0, q_22= -3. Written in symmetric form:

B=R1 [1,2]
R2[2,-3]

Halving the sum of q_12 and q_21

Question 16

Deriving Canonical form

Answer 16

Since B is symmetric it has those properties:

B=PDP^T where P is an orthogonal matrix of columns of eigenvectors
And D us diagonal of eigenvalues.

Hence we can diagonalisa the quadratic form Q(x) =x^T B x = x^T PDP^T x = ~x^T D ~x

Where ~x = P^Tx is a vector ~x = (u_1•x, u_2•x,…, u_n•x) columns vector

Hence Q(x) = sum of eigenvalues * ~x ^2 for each ~x_i up to n. You

Question 17

Canonical form

Answer 17

The quadratic form Q(x)?= x^T B x =
( y_1 ^2 +… + y_r ^2) - ( y_(r+1) ^2 +…+ y_(r+s)^2)

Where y_i = v_i• x, are linear combinations of x_i.
Eigenvalues λ_i are ordered:

The first r are positive, the next s are negative and the rest 0 ( In total all n)

v_i =
• sqrt( λ_i) u_i if i is less than or equal to i.e. For λ_i positive
• sqrt( -λ_i) u_i if I is bigger than or equal to r+1 and less than or equal to r+s
I.e. For λ_i is negative
• u_i if i in [r+s,n] those λ_i=0

Question 18

Rank and signature of quadratic forms

Answer 18

The rank of Q is r+s and signature is r-s

For canonical form

That is rank is the number of positive and negative eigenvalues, non zero

r-s is the difference between the two with number of positive taken negative (how much more positive)

Question 19

a quadratic form is positive definite, negative definite, semi definitive def

Answer 19

Positive definite: positive for all x not equal to 0.
Negative definite/ negative for all x not equal to 0.
Positive semidefinite: Q(x) is bigger than or equal to 0 for all x not equal to 0
Negative semidefinite: Q(x) is less than or equal to 0 for all x not equal to 0

INDEFINITE IF Q(x) can take both positive and negative values

HENCE WE CAN DETERMINE THE NATURE OF THE QUADRATIC FORM by determining eigenvalues of symmetric matrix B

Distance squared function has rank n and signature n

Question 20

a quadratic form is positive definite, negative definite, semi definitive relation with eigenvalues

Answer 20

Positive definite if all positive eigenvalues

Negative definite if all negative eigenvalues

Becomes semidefinite if includes some =0

Indefinite if at least one positive one negative eigenvalues

Question 21

Example determine the canonical form of the quadratic form
Q(x) = 5x_1^2 + 6x_2^2 + 7x_3^2 -4x_1x_2 + 4x_2x_3

Where x = (x_1, x_2,x_3)^T what is the rank and signature

Answer 21

We can write as
Q(x) = x^T B x

Where B is the symmetric matrix

[5 , -2, 0]
[-2, 6, 2]
[0, 2, 7]
The corresponding eigenvalues and eigenvectors can be normalised.we check these are an orthonormal set.

3 and (1/3) (2,2,-1)
6 and (1/3)(2,-1,2)
9 and (1/3)(-1,2,2)
The canonical form is therefore Q(x) =y_1^2 + y_2^2 + y_3^2 
Where y_i =v_i •x and hence
y_1 = (√3/3)(2x_1 +2x_2-x_3)
y_2 = (√6/3)(2x_1 -x_2+2x_3)
y_3 = (3/3)(-x_1 +2x_2+2x_3)

Q(x) is therefore positive definite with rank 3 and signature 3

Question 22

Hessian matrixDeriving

Answer 22

Determine the nature of stationary points of a function of more than two variables

If x in R^2 and if f(x_1, x_2) has continuous first and second derivatives with stationary point at (a,b). Then the Taylor series …
f(a+h, b+k) = f(a,b) + 0.5 ( f_11 h^2 + 2f_12 hk + f_22 k^2) +…
In matrix form
f(a+h, b+k) = f(a,b) + 0.5 (h k) H col_vector(h,k)
H = hessian = [f_11, f_12]
[f_21, f_22]

Symmetric as second partial derivatives same

(In addition stationary points able to determine by eigenvalues)

Question 23

Stationary points and hessian

Answer 23

For f(x), x∈R^n that has a stationary point at x=x_0

f(x_0 + δx) = f(x_0) +0.5(δx)^T H (δx) +…
= f(x_0) + 0.5 Q(δx) +…

Where Q(δx) is a quadratic form associated with the symmetric matrix H whose components ate partial second derivatives of f evaluated at x=x_0

h_ij = (∂^2 f/ ∂x_i ∂x_j. ) (x_0) for I,j = 1,…,n

The stationary point is:
A LOCAL MINIMUM if the quadratic form Q(δx) is positive definite.

A LOCAL MAXIMUM if the quadratic form Q(δx) is negative definite.

Q(δx) is indeterminate then stationary point is a saddle

If Q(δx) is either positive or negative semidefinite then further investigation in order to to determine its nature

Question 24

Stationary points and hessian eigenvalues

Answer 24

If a function f(x) for x in R^n that has CONTINUOUS FIRST AND SECOND derivatives has a stationary point at x=x_0 and if the HESSIAN H at x_0 has eigenvalues λ_1,…, λ_n

• x_0 is a local minimum if λ_i is bigger than 0 for I=1,…,n
• x_0 is a local maximum if λ_i is less than 0 for I =1,…,n
• x_0 is saddle point if at least one eigenvalues is positive and at least one negative AND NONE 0

• if any at all λ_i =0 then we require further investigation

Question 25

Example find and determine nature of stationary points of the function f(x,y) = x^2 -4xy + 3y^2 -2x +y -2

Answer 25

Finding partial derivatives for the hessian:
f_xx =2
f_xy=-4
f_yy=6
f_yx=-4

Hessian is [2,-4]
[-4, 6]
Has characteristic equation giving eigenvalues which have DIFFERENT SIGNS and thus the stationary point is a saddle point

Question 26

Example find and determine nature of stationary points of the function f(x,y,z) = 2x^2 -2xy + 3y^2 + z^2 +4xz

Answer 26

Has stationary point at f_x = f_y = f_z =0

f_x = 4x -2y + 4z
f_y = 6y - 2x
f_z =2z +4x

Hence stationary point only at (0,0,0)

Has hessian matrix [4,-2,4]
[-2,6,0]
[4,0,2]

Looking at characteristic equation product gives a negative value therefore they are either all negative or only one is negative.

Sum is a positive value and hence they can’t all be negative. Hence we have 2 positive and one negative eigenvalue. It has rank 3 and signature 1 and the point is a saddle point