lecture 6 - predictive modeling with time series

Question 1

stationarity

Answer 1

• time series require a form of stationarity

stationary if
1. trends and periodic variations are removed. i.e., it has no trends. mean and variance do not change over time
2. variance of the remaining residuals are constant over time. i.e., fluctuations around the mean are uniform over time
3. lagged autocorrelation should remain constant

Question 2

time series focus on:

Answer 2

1. understanding periodicity and trends
2. forecasting
3. control

Question 3

time series can be decomposed into these components

Answer 3

1. periodic variations (daily, weekly, etc.)
2. trend (how the mean evolves over time)
3. irregular variations (left after we remove periodic variations and trend)

Question 4

why stationarity

Answer 4

simplifies the model building process

Question 5

lagged auto correlation

Answer 5

• additional criterion for time series
• represents in how far there is a correlation between a time series and a shifted version of itself (with λ time steps)

Question 6

way to get rid of trends (to reach stationarity)

Answer 6

1. apply filter/smoothing to data
   - assume time series of values x_t with a fixed step size Δt
2. remove a trend

Question 7

apply filter to data

Answer 7

1. taking q points in the future and past into account. this generates a new time series z_t
2. choosing the a weight

Question 8

filtering weight: triangular shape

Answer 8

this a weights points based on their distance from r

• choose this if measurements closer to t are more important
• measurements closer to t get more weight

Question 9

filtering weight: moving average

Answer 9

inverse of (2q + 1)

Question 10

filtering weight: exponential smoothing

Answer 10

exponentially decreases the weight as you move away

• choose this when mostly past points are important

Question 11

filtering weight: q parameter

Answer 11

determines how many points before and after the current point are considered in the smoothing process

Question 12

removing a trend

Answer 12

• z_t = the difference between the current and previous measurement
• with differencing x_{t} and x_{t-1}, we remove the linear trend component, making the series stationary
• the idea is that is there is a trend in the data, it will affect x_{t} and x_{t-1} similarly. by subtracting one from the other, the trend component is eliminated, leaving behind fluctuations that are more stationary
• we can apply this differencing operator d times for more complex trends

Question 13

removing a trend: if x_{t-1} does not give a good estimation of the trend

Answer 13

we can use an exponential smoothing z_t and take x_t - z_t.

Question 14

learning algorithms with time

Answer 14

1. ARIMA
2. NNs with time
   
   • RNN
3. deep learning
   
   • LSTM
   • TCN
4. echo state

Question 15

ARIMA components

Answer 15

1. probability distribution: assume that measurements are generated by a probability distribution P_t at each time point t
2. expected mean mu(t) of the distribution at time t. this represents the central tendency of the time series at any point
3. auto-covariance function (gamma(t1,t2)): measures the covariance of the time series at two different times.

Question 16

ARIMA goal

Answer 16

estimate P_t based on previous values for this distribution

• P_t = probability distribution of measurements at time point t

Question 17

ARIMA: when is a series stationary

Answer 17

1. when the mean is constant
2. when the autocovariance only depends on the time difference λ = t2 - t1

Question 18

ARIMA: W_t parameter

Answer 18

• represents the noise we encounter
• we can account for this noise by a moving average component (with q past values)

Question 19

ARIMA: d parameter

Answer 19

differecing with order d

• ARIMA removes the trends with this parameter

Question 20

ARIMA: p & q parameters

Answer 20

number of steps we’re looking back

• p: for measurement at time t
• q: for noise

Question 21

ARIMA: finding parameter values

Answer 21

1. p: look at correlation between x_t and x_{t-p}. i.e., a partial correlation function
2. q & d: grid search, then determine goodness of fit
3. other parameters: use past data to optimize weights, autoregressive component, etc.

Question 22

simple recurrent neural network

Answer 22

• neural network with time (i.e., designed to handle sequential data)
• include cycles that allow information to persist (memory)

1. forward path: input to output
2. once prediction is made, the error is propagated back into the network. this allows you to update the weights

Question 23

RNN: backpropagation through time

Answer 23

• cycles of the simple RNN make training complex as the output at a given time step depends on previous computations
• this is fixed by unfolding the network by creating an instance of the network for each previous time point and connecting these
• this combined network is without cycles

Question 24

RNN: forward paths calculation

Answer 24

• weight update rule = learning rate * error term at time t * predicted output of node j at time t
• error term depends on if node j is an output node or hidden node

Question 25

RNN: recurrent connections calculation

Answer 25

• the weight update rule is similar to forward connections, but includes a temporal component
• weight update rule = learning rate * error term at time t * predicted output of node j at time t-1
• the error term formula is similar to the hidden node case but considers the error propagation over time

Question 26

LSTM networks

Answer 26

• have longer memory than RNNs
• solve the vanishing gradient problem, which allows them to learn long-term dependencies more effectively
• internal cell state C (vector of numbers) is updated based on gates (i.e., our memory)

Question 27

LSTM: internal state C & gates

Answer 27

• C represents memory

1. forget gate: Decides which information to discard from C (i.e., each element in the vector)
2. addition gate: controls which information to add to the cell state

(1) candidate values C_t: Determines what new information could be added to the cell state. (i.e., how to update)
(2) update magnitude i_t - Determines how much of the new information to add.

3. updating cell state: Combines the previous cell state C_{t-1} (scaled by the forget gate) with the new candidate values (scaled by the input gate).
   - Combines the forget gate and input gate effects to update the cell state.
4. output gate: new prediction based on previous prediction, cell state, and current input.
   - Controls what information from the cell state to output as the hidden state.

Question 28

temporal convolutional networks (TCN)

Answer 28

• network based on convolutions
• outperforms LSTM networks and have better long term memory
• sequence to sequence

Question 29

TCN: causal connections

Answer 29

Only past information is used to predict future values, ensuring the model doesn’t look ahead in time.

Question 30

TCN: sequence to sequence

Answer 30

1. Equal Lengths: Input and output sequences are of equal length, with zero padding used if necessary.
2. Uniform Layer Size: All layers in the network have the same size.

Question 31

TCN: features - dilations and kernel size

Answer 31

1. kernel size k expresses how many values we are considered in a layer
2. dilation factor d expresses how detailed we consider the timesteps to come to k values
   - Higher d: Larger gaps, capturing a broader range of history with fewer layers needed.
   - Lower d: Smaller gaps, capturing more detailed local information.
   - d exponentially increases over layers (d = O(2^i))

Question 32

TCN: exponential increase of d

Answer 32

1. guarantees that each input is used
2. captures a bigger and bigger history

Question 33

TCN: residual blocks

Answer 33

• Instead of simple convolutional layers, TCNs use residual blocks, which are structures that help the network learn more effectively
• The residual block typically includes elements like dropout (to prevent overfitting), activation functions (e.g., ReLU), and normalization steps (e.g., WeightNorm).
• The key benefit of residual blocks is that they help address the problem of vanishing gradients

Question 34

echo state networks

Answer 34

• ESNs simplify complexity of RNNs by having a reservoir of neurons with fixed, random connections. This reduces the number of trainable parameters.
• reservoir can have cycles to allow for a memory component

Question 35

echo state network: weights

Answer 35

• not all the weights are trained, but are set randomly

1. input weights and weights in the reservoir are set randomly (and not changed during the process)
2. weights from reservoir to output are the only ones that are being trained

Question 36

echo state networks: reservoir

Answer 36

• if you have a big enough reservoir, you can create all kinds of variants of the signal over time as features
• then you just need to pick up the right features in the output layer in order to decide how to predict for the next time step

Question 37

echo state networks: learning

Answer 37

• learning a W_{out} that minimizes the difference between the actual and predicted y
• since we only train W_{out} its not hard to train this model, given that the reservoir has enough richness

Question 38

echo state networks: output

Answer 38

1. state of reservoir at i+1 = reservoir activation function( input weightsx_{t+1} + reservoir weights state of reservoir at i )
2. predicted output at i+1 = output layer activation function( output weights * state of reservoir at i+1)

Question 39

echo state networks: echo state property

Answer 39

• tells us how to create a random reservoir that has useful signals
• meaning that the effect of a previous state (r_i) and a previous input (x_i) on a future state (r_i+k) should vanish gradually as time passes (k –> inf), and not persist or get amplified
• The echo state property ensures that the influence of a previous state or input on future states diminishes over time. This prevents the amplification of past inputs and maintains stability in the network.

Question 40

memory capacity, worst to best:

Answer 40

1. RNN
2. LSTM
3. TCN