Lecture 19

Question 1

Correlation coefficient

Answer 1

line fit.
di = yi-yfit
a = Σ(di)^2
b = Σ(yi-ymean)^2
rvalue = √(1-a/b) correlation coefficient
R = 1 → good fit

Question 2

Linear least squares = line ?

Answer 2

No, linear in its coefficient !

OK if a vandermonde matrix can be written.

Question 3

Fitting a line {{t1, y1}, …, {tm, ym}}

Answer 3

Overdetermined system Ax = b ([[t1 1],…,[tm 1]] [a b] = [[y1],…[ym]]), A mxn where m>n
In most cases, no exact solution because b∉range(A) and Ax≅b

Question 4

Least squares

Answer 4

A mxn with m>n. Find x minimising residual r=b-Ax
||Ax-b||_2^2 = (Ax-b).T@(Ax-b)
∇ = 2A.T@(Ax-b) = 0
so A.T@A@x = A.T@b (normal equations)
Minimum because Hessian=2A.T@A is positive definite (symmetric, eigenvalues >= 0)

Question 5

A matrix least squares

Answer 5

Vandermonde matrix
1 t1 t1^2 ...
1 t2 t2^2 ...
... ... ...
1 tm tm^2 ...

Question 6

Residual

Answer 6

r= ||Ax-b||_2 = ||yfit-y||_2

Question 7

y = c1 + c2.sin(tΠ) + c3.sin(tΠ/2) + c4.sin(tΠ/4)
Value A_{23} ?
ti yi
.5 .72
1 .79
1.5 .72

Answer 7

sin(t2.Π/2) = sin(Π/2) = 1

Question 8

Linear least squares solution?

Answer 8

Always

Question 9

Columns of A linearly independant

Answer 9

rank(A) = n and solution of normal equations unique (A.T@A only positive eigenvalues), otherwise rank(A) less n, A rank-deficient (A.T@A singular) and solution not unique

Question 10

Cost solve normal equations

Answer 10

A.T@A@x=A.T@b
cost A.T@A = mn^2
cost A.T@b = mn
factorization A.T@A n^3 (LU 2/3n^3, cholesky 1/3n^3)
solve with factorization n^2
Overall cost mn^2 (> n^3 because m>n)