Lecture 20

Question 1

Data fitting, y=Ax in general ∈ ?

Answer 1

∈ range(A), r orthogonal to all columns of A (y closest to b ∉ range(A)) Indeed, A.T@r = 0 => A.T@A@x = A.T@b

Question 2

Condition number for normal equations

Answer 2

cond(A.T@A) = (cond(A))^2, worsen the conditioning of the matrix. Solution: use SVD (even if Cholesky is less costly!)

Question 3

SVD to solve a square system Ax=b? (A nxn)

Answer 3

A = UΣV.T

1) Solve Σy=U.T@b (y=V.T@x), we get yi=(ui.b)/σi
2) Solve x=Vy = y1.v1 + … + yn.vn, we get x = Σ_{i=1}^n yi.vi

Question 4

Shapes U, Σ, V of the reduced SVD of A 10x14 matrix?

Answer 4

U 10x10
Σ 10x10
V 14x10

Question 5

Solution reduced SVD full rank least square problem

Answer 5

A = Ur@Σr@V.T, σi≠0
A.T@A@x = A.T@b
gives x=V@Σr^{-1}@Ur.T@b (unique solution)

Question 6

Solution reduced SVD rank deficient least square problem

Answer 6

rank(A) < n. Find min_x ||Ax-b||_2, min ||x||2.
A = Ur@Σr@V.T
1) Solve Σr@y = Ur.T@b with y=V.T@x, we get yi=(ui.b)/σi if σi≠0, 0 otherwise.
2) x=V@y = Σvi.yi = Σ{σi≠0} vi.(ui.b)/σi, that is x = V@Σr^+@Ur.T@b

Question 7

Cost of solving min_x ||b-Ax||_2^2 (A=UΣV.Tx calculated!)

Answer 7

O(mn) indeed, x=V@Σr^+@Ur.T@b (there’s also a n^2 operation but m>n so mn > n^2)

Question 8

Rank(A) necessary to solve least squares?

Answer 8

Rank(A) must be > number of coefficients, otherwise bad results.

Question 9

Python eigenvalues, eigenvectors

Answer 9

eigvals, eigvecs = np.linalg.eigh(A)

Question 10

Python svd

Answer 10

U, S, Vt = np.linalg.svd(A, full_matrices=False)

False for reduced SVD. Be careful, Vt returned and S is a 1D array! (use np.diag)

Question 11

Python least squares

Answer 11

solution, residual, rank, σiOfA = np.linalg.lstsq(A, b)

Question 12

Python linear/poly reg

Answer 12

from scipy import stats
slope, intercept, rvalue, pvalue, stderr = stats.linregress(x, y)
cubic, quadratic, linear, intercept = np.polyfit(x, y, 3)

Question 13

Cholesky factorization, why?

Answer 13

Because the system of normal equations is symmetric, the normal equations can be solved by computing and using a Cholesky factorization

Question 14

Solve non-linear least-squares

Answer 14

Use of steepest descent