Identification of ARMA processes

Question 1

Identification of AR models

Answer 1

1) data set

2) model class: AR model in canonical representation
m(θ) : y(t) = 1/A(z) * e(t)

3) performance index JN(θ)
it is a quadratic function of θ

4) optimization (type 1)
an explicit unique solution exists

Question 2

Identification of ARMA models

Answer 2

1) data set

2) model class: ARMA model in c.r.
m(θ) : y(t) = C(z)/A(z) * e(t)

3) computation of performance index
in general it is not a quadratic function of θ and not a convex function

4) optimization (type 3)
iterative method starting from an initial guess

Question 3

how to perform a single iteration in the optimization

Answer 3

• build a local quadratic approximation of JN(θ) around θi
• find the minimum point of this quadratic function θi+1
• repeat
• a (local) minimum is found at the end

This is the Newton method, and it is much faster than the gradient method.
With quasi-Newton methods, the convergence properties improve

If the performance index is not a convex function of θ, an empirical solution can be used:

• start the procedure many times, from random different initial conditions
• find many global minima
• choose the best as the global minimum

Question 4

On the model order selection

Answer 4

• general idea: try them all:
  
  • consider a set of balanced ARMA models, from ARMA(1,0) to ARMA(M,M)
  • M is usually much smaller than N
  • for each model, perform the identification procedure
  • compute the value of the performance index at the optimum JN(θ^N, nθ)
  • take the model with the minimum performance index
• problem: the performance index at the optimum, as a function of the number of parameters, is a monotonically decreasing function, so with this approach the maximum order model is Always chosen
• > this problem is known as overfitting

Question 5

how to solve the overfitting issue

Answer 5

The most used approach is the cross-validation:

• the data set is split into two parts: training set, used for system identification, and validation set, used for checking the generality of the model
• for all the model classes, find the optimal θ^N/2 by minimizing the performance index, using only the training set
• compute the values of the performance index at the optimum, for both training and validation sets
• > JN/2(θ^N/2; nθ; phiT)
• > JN/2(θ^N/2; nθ; phiV)
• it happens that, while the values for the performance index with the training set decrease with increasing order, the values of the optimal performance index with the validation data set have a minimum
• the best order of the model class correspond to this minimum
• for models with higher order, the overfitting problem occurs, and the model is fitting also the noise contained in the specific data set

Question 6

Why a restriction to AR or MA models can be useful?

Answer 6

In some cases, the model class can be selected as AR or MA models:

• AR models are very quick and easy to be designed, so the identification procedure requires low computational power, low cost, low power hardware
• MA models are a priori as. stable