Optimization Class

Question 1

HW 1 - Question 1. Three funds. Total of $12,000. Muni = 7%, CD = 8%, High Risk = 12%. Constraints: No more than 2K in High Risk Three times more in Muni than CD What are optimal investments for each fund?

Answer 1

Objective Function: Min - (r x+r y+r z)

x = muni, y = CD, z = High Risk

Constraints: z

Question 2

Understanding the Least Squares (data fitting)

Answer 2

The best fit in the least squares sense minimizes the sum of squared residuals. Residuals are the difference between an observed value and the fitted value provided by the model. Hence - least squares solution is obtained by minimizing the sum of squared distances between the line and each point.

Question 3

Normalized Vector

Answer 3

The normalized vector of X is a vector in the same direction but with norm (length) 1. It is denoted X^ and given by
X^^=(X)/(|X|),

where |X| is the norm of X. It is also called a unit vector.

Question 4

What is Norm?

Answer 4

Norm of a mathematical object is a quantitly that in some (possibly abstract) sense describes the length, size and extent of the object.

Most commonly, the unqualified term “norm” refers to the flavor of vector norm technically known as the L2-norm. This norm is variously denoted ||x||₂, ||x||, or |x|, and gives the length of an n-vector x=(x₁,x₂,…,x_n). It can be computed as

|x| = sqrt(x₁² + x₂² +…+ x_n²)

Question 5

While Loop - Matlab

Answer 5

While loop runs until certain condition is met. While loops can continue indefinitely. Very useful if number of iterations are not known.

Sample Code: While – {Logical Expression}

{code}

end.

If the Logical expression is true, it will run the code. If the logical expression is not met, then the code will exit. In the code, something inside the “while” loop must change otherwise the loop will never exit or execute.

The symbols &, |, and ~ are the logical array operators AND, OR, and NOT.

Question 6

Cardinality of Matrix

Answer 6

The Cardinality of a set is is a measure of the “number of elements of the set.” For example, the set A = {2, 4, 6} contains 3 elements, and therefore A has a cardinality of 3.

Question 7

First Order Method for Sparse Covariance selection

maximize log det X − 〈Σ, X〉 − ρ Card(X)

**subject to ** αI_n ≼ X ≼ βI_n

Answer 7

Zeros in the inverse covariance matrix corresponds to conditionally independent variables in the model. This approach can be used to simultaneously determine a robust estimate of a covariance matrix.

In this paper, they are trying to strike a trade-off between maximality of the likelihood and the number of non-zero elements in the inverse covariance matrix. Their approch is particular useful at discovering structure - precisely conditional independence properties in data.

Question 8

Trace of Matrix

Answer 8

Trace of a matrix is the sum of its diagonal elements

Question 9

Tridiagonal matrix

Answer 9

a matrix that has nonzero elements only on the main diagonal, the first diagonal below this, and the first diagonal above the main diagonal.

Question 10

Rank of a Matrix

Answer 10

The column rank of a matrix A is the maximum number of linearly independent column vectors of A. The row rank of A is the maximum number of linearly independent row vectors of A.

Question 11

Kernel of a Matrix

Answer 11

In linear algebra, the kernel or null space (also nullspace) of a matrix A is the set of all vectors x for which Ax = 0. The kernel of a matrix with n columns is a linear subspace of n-dimensional Euclidean space. The dimension of the null space of A is called the nullity of A.

Question 12

Orthogonal Matrix and Vector

Answer 12

Two vectors are orthogonal iff their scalar product is zero.

An Orthogonal matrix is if: U^TU = I

Question 13

Cauchy-Schwartz inequality:

Answer 13

x^Ty < = ||x||₂ · ||y||₂

Question 14

Test Set and Training Set

Answer 14

• A test set is a set of data used in various areas of information science to assess the strength and utility of a predictive relationship.
• A test set is a set of data that is independent of the training data, but that follows the same probability distribution as the training data. If a model fit to the training set also fits the test set well, minimal overfitting has taken place. If the model fits the training set much better than it fits the test set, overfitting is likely the cause.

Question 15

Regularization

Answer 15

• This refers to a process of introducing additional informatin in order to solve an ill-posed problem or to avoid overfitting. This information is usually of the form of penalty for complexity, such as restrictions for smoothness or bounds on the vector space norm.
• In stat and machine learning, regularization methods are used for model selection - in particular to prevent overfitting by penalizing models with extreme parameter values.
• L1 and L2 regularization - which can be added to learning algorithms that minimizes a loss function E(X,Y) by instead minimizing E(X,Y) + Alpha||w|| (where w is model’s weight vector)