Chapter 6: Statistical Models for Time Series

Question 1

How do linear models for time series differ from models applied to cross-sectional data?

Answer 1

They account for the correlations that arises between data points in the same time series, in contrast to the standard methods applied to cross-sectional data, in which point is assumed to be independent.

Question 2

What assumptions would allow ordinary least squares to be applied to time series data?

Answer 2

With respect to time series behaviour:
- Time series has a linear response to its predictors.
- No input variable is constant over time or perfectly correlated with another input variable.

With respect to error:
- For each point in time, the expected value of the error, given all explanatory variables for all time periods is 0.
- The error at any given time period is uncorrelated with the inputs at any time period in the past or future.
- Variance of the error is independent of time.

Question 3

What is an autoregressive model (AR)?

Answer 3

AR models rely on the intuition that the past predicts the future and so posits a time series process in which the value at a point in time t is a function of the series’s values at earlier points in time.

Question 4

What is the equation for the simplest AR model, AR(1)?

Answer 4

y(t) = b0 + b1 * y(t-1) + e(t)

Question 5

Does a time series need to be stationary to be modelled using AR?

Answer 5

Yes, the time series does need to be stationary.

Question 6

What is the difference between strong and weak stationarity?

Answer 6

Weak stationarity only requires the mean and the variance of a process to be invariant.

Strong stationarity requires the distribution of the random variables output by a process to remain over time. It demands the statistical distribution of y1, y2, y3, y4 to be the same as y101, y102, y103.

Question 7

What is the definition of a distribution?

Answer 7

A distribution is a statistical function describing the probabilities for all possible values that a particular value will be generated by a process.

Question 8

What is Akaike Information Criterion (AIC) for a model?

Answer 8

AIC = 2k - 2lnL, where k is the number of parameters and L is the maximum likelihood value for a that function. In general we want to lessen the complexity of the model (lessen k) but increase the goodness of fit (i.e L), so we favour models with lower AIC’s than higher.

Question 9

What is a likelihood function?

Answer 9

A likelihood function is a measure of how likely a particular set of parameters for a function is in relation to the other parameters for that function given the data.

For example, when fitting a linear model to y = [1,2,3], x = [1,2,3] using y = b * x, your likelihood function would tell you that an estimate of b = 1 was far more likely than an estimate b = 0.

Question 10

REFACTOR THIS CARD
What check should we do after fitting an AR(p) model to assess goodness of fit with respect to the models errors?

Answer 10

Plot an ACF of the residuals (error) at each lag to see if any cross the significance threshold.

Question 11

If we see significant autocorrelation between the errors in a model, what should we do?

Answer 11

Return to the model and consider additional terms to add complexity to account for the significant autocorrelation of the residuals.

Question 12

What is the Lijung-Box test?

Answer 12

The Lijung-Box test is an overall test of the randomness of a time series. It poses the following:
- H0: The data does not exhibit serial correlation.
- H1: The data does exhibit serial correlation.

Question 13

When is the Lijung-Box test applied?

Answer 13

The test is commonly applied to AR and ARIMA models, more specifically to their errors rather than the models themselves.

Question 14

When using an AR model to predict, how should we assess the performance of the predictions?

Answer 14

We should like a the correlation between the raw time series and predictions, but more importantly the correlation between the difference time series and predicted time series.

Question 15

What are Moving Average (MA) models?

Answer 15

A moving average model is similar to an autoregressive model except that the terms included in the linear equation refer to present and past error terms rather than present and past values of the process itself.

Question 16

How do you determine the parameters to use for a MA model?

Answer 16

Use an ACF to determine the order of the MA process.

Question 17

How would you use the output of an ACF in a MA model?

Answer 17

Where there are significant values at lags, fit an MA model with these lags.

Question 18

What does ARIMA stand for and what does it model?

Answer 18

AutoRegressive Integrated Moving Average. This combines AR and MA models and accounts for differencing.

Question 19

How do you determine which model out of AR, MA or ARMA best describe the time series?

Answer 19

ACF behaviour:
AR(p) - ACF falls off slowly.
MA(q) - Sharp drop after lag = q.
ARMA - No sharp cut off.

PACF behaviour:
AR(p) - Sharp drop off after lag = p.
MA(q) - Falls off slowly.
ARMA - No sharp cut off.

Question 20

What is Wolds theorem?

Answer 20

Wolds theorem tells us that every covariance-stationary time series can be written as the sum of two times, one deterministic and one stochastic.

Question 21

What are reasonable values of p, d, in the context of ARIMA(p, d, q)?

Answer 21

A practitioner notes that one should be skeptical of:
d > 2
p > 5
q > 5
p&raquo_space; q | p &laquo_space;q

Question 22

What is ARIMA(0, 0, 0)?

Answer 22

ARIMA(0, 0, 0) is a white noise model.

Question 23

What is ARIMA(0, 1, 0)?

Answer 23

ARIMA(0, 1, 0) is a random walk.
Adding a non-zero constant to this is called a random walk with drift.

Question 24

What is ARIMA(0, 1, 1)?

Answer 24

This is an exponential smoothing model. ARIMA(0, 2, 2) is the same as Holts linear method, which is exponential smoothing for data with an underlying trend.

Question 25

What is the Box-Jenkins heuristic for choosing parameters for an ARIMA model?

Answer 25

1. Use your data, visualizations, and underlying knowledge to select a class of model appropriate to your data.
2. Estimate the parameters given your training data.
3. Evaluate the performance of your model based on your training data and tweak the parameters of the model to address the weaknesses you see in the performance diagnostics.

Question 26

When inspecting the PACF of residuals for an ARIMA model, what do large values suggest?

Answer 26

Large PACF values for residuals suggest that our model does not fully describe the auto-regressive behaviour of the time series. If this is the case, we add a higher order AR component and re-run the model.

Question 27

What is a quick way to check how well forecasts do?

Answer 27

A correlation between the forecasted values and the observed values.

Question 28

What is an alternative to manually fitting models to determine their parameters?

Answer 28

Using automated model selection based off information loss criteria such as AIC.

Question 29

What is Vector Autoregression (VA)?

Answer 29

Vector Autoregression generalises an AR(p) model to account for multiple times to capture how they relate to each other, i.e. can one predict the other.

Question 30

What is seasonal ARIMA (SARIMA)?

Answer 30

A SARIMA model assumes multiplicative seasonality and postulates that the seasonal behaviour itself can be thought of as an ARIMA process.

Question 31

What are ARCH, GARCH family of models?

Answer 31

Autoregressive Conditional Heteroskedasticity (ARCH) models used for times that don’t have constant variance (i.e. stock prices) and that the variance appears autoregressive conditional on the earlier variances (for example high volatility days on the stock exchange occur in clusters).

In these models, the variance of a process is modelled rather than the process itself.

Question 32

What are the advantages of statistical models of time series?

Answer 32

• These models are simple and transparent, so they can be understood clearly in terms of their parameters.
• Because of the simple mathematical expressions that define these models, it is possible to derive their properties of interest in a rigorous statistical way.
• You can apply these models to fairly small data sets and still get good results.
• These simple models and related modifications perform extremely well, even in comparison to very complicated machine learning models. So you get good performance without the danger of overfitting.
• Well-developed automated methodologies for choosing orders of your models and estimating their parameters make it simple to generate these forecasts.

Question 33

What are the disadvantages of statistical models of time series?

Answer 33

• Because these models are quite simple, they don’t always improve performance when given large data sets. If you are working with extremely large data sets, you may do better with the complex models of machine learning and neural network methodologies.
• These statistical models put the focus on point estimates of the mean value of a distribution rather than on the distribution. True, you can derive sample variances and the like as some proxy for uncertainty in your forecasts, but your fundamental model offers only limited ways to express uncertainty relative to all the choices you make in selecting a model.
• By definition, these models are not built to handle nonlinear dynamics and will do a poor job describing data where nonlinear relationships are dominant.