Statistical modelling in Space and Time

Question 1

What variables is a Gaussian process dependent on/defined by?

Answer 1

A mean function

A covariance function (as a function of the distance between two points at x1 and x2)

Question 2

Do Gaussian processes model space or time?

Answer 2

Spatial fields

Question 3

What is a strictly stationary process? (stochastic processes)

Answer 3

A strictly stationary process has the same statistical properties everywhere.
Denote the process by Y. Then the distribution of Y(x) is the same as for Y(x + h).

Question 4

Define a deterministic function. What is the difference between a deterministic and a stochastic process?

Answer 4

A function is considered deterministic if it always returns the same result set when it’s called with the same set of input values.

In deterministic models, the output of the model is fully determined by the parameter values and the initial conditions. Stochastic models possess some inherent randomness. The same set of parameter values and initial conditions will lead to an ensemble of different outputs.

Question 5

Define a weakly stationary process

Answer 5

For a weakly stationary process the first (mean) and second order moments are the same everywhere, and the covariance simply depends on distance.

This means the process has the same mean at all time points, and that the covariance between the values at any two points, x and x+h, depend only on h, the distance between the two points, and not on the location of the points in the region.

(w.l.o.g assume mean is 0)
( Cov(x1, x2) = Cov(x1 + h, x2 + h) )

Question 6

What is another term used for Weak Stationarity?

Answer 6

second order stationarity

Question 7

Does Strict stationarity imply weak stationarity?

Answer 7

Yes
In general the converse does not apply but it does for Gaussian processes; and only for Gaussian processes which are the only stochastic processes defined solely by their first and second moments.

Question 8

Define a Gaussian Process.

Answer 8

A Gaussian process is an infinite dimensional (continuous) stochastic function/process all of whose marginal, conditional and joint distributions are Gaussian.

Question 9

What are the first and second order moments of a (stochastic) process?

Answer 9

The first moment of xᵢ is the expected value/mean
E[xᵢ]

The second moment of xᵢ is the expected value of xᵢ²
E[xᵢ²]

Question 10

What is intrinsic stationarity?

Answer 10

Assume a constant mean process. Then
E[Y(x + h) − Y(x)]² = Var[Y(x − h) − Y(x)] = 2γ(h)
If this only depends on h then the process is said to have intrinsic stationarity

Question 11

What property does the weakly stationary process’s covariance function have? Proof.

Answer 11

Covariance function has to be positive definite.

Weakly stationary, therefore the second moment of xᵢ is finite for all t;
i.e. ∀t, E[xᵢ²] < ∞
Which also implies E[(xᵢ-𝜇)²] = Var(xᵢ) < ∞; i.e. that variance is finite for all t)

(CARD NOT FINISHED)

Question 12

The Wiener-Khinchin (or Khinchine) theorem is a special case of which theorem for time series?

Answer 12

Bochner’s Theorem

Question 13

Define an isotropic process

Answer 13

If the covariance depends only on distance (not direction) the process is isotropic.
For an isotropic process the covariance function is univariate (involving one variable quantity).

Question 14

Separable process

Answer 14

In 2D, the correlation structure in the x direction does not change with y (and vice versa).
This holds for multivariate extensions …

Question 15

Difference between a Gaussian distribution and Gaussian Process

Answer 15

Gaussian distribution is defined by its mean and variance, whereas a Gaussian process is defined by a mean function and a covariance function (positive definite).

Question 16

Weierstrass Theorem

Answer 16

By increasing the order of a polynomial we can fit any smooth function to arbitrary precision

Question 17

Gaussian process with Matérn 3/2 covariance possesses how many derivatives?

Answer 17

One

Question 18

Via Bochner’s theorem the associated spectral density of a Matérn covariance function is the pdf of what distribution?

Answer 18

t-distribution

Question 19

Define a nugget

Answer 19

an independent (iid normally distributed) error added to each data point

Question 20

Three reasons why you would add a nugget?

Answer 20

1) Instrumental error (often small) - data isn’t entirely smooth because of instrumental error
2) Small scale variation that we don’t want to model by the Gaussian Process - concerned with the larger scale stuff
3) Sometimes we add nuggets for numerical reasons to prevent the covariance matrix being too smooth - guarantees the matrix will be positive definite and numerically stable.

Question 21

Name the layers of the data when considering it in a hierarchical way (model).

Answer 21

Data layer
Process layer
Parameter layer

Question 22

What layer is missing from the Empirical Hierarchical model and why?

Answer 22

Parameter layer

The parameters θ are fixed numbers

Question 23

Bayesian Hierarchical model steps

Answer 23

Put a prior distribution on the θ (length scale)
And use conditional distributions to find the distribution of Z (the data).
Bayes theorem then allows to ‘reverse’ the hierarchy.

Question 24

Another name for variogram

Answer 24

Semi-variogram

Structure function

Question 25

Equation for the variogram

Answer 25

γ(h) = σ² − C(h) Given a covariance function we can calculate the corresponding variogram

Question 26

Assume the process is ergodic, what can consequently be concluded about the covariance function, and hence the variogram?

Answer 26

C(h) → 0 as h → ∞ lim h→∞ [γ(h)] = σ² And we have C(h) = σ² − γ(h) = lim k→∞ [γ(k)] − γ(h)

Question 27

Define kriging

Answer 27

Kriging is using the variogram to interpolate between data | i.e. using a variogram instead of a covariance function

Question 28

Define simple kriging

Answer 28

In simple kriging the mean is assumed to be constant and known (i.e. zero)

Question 29

Define Ordinary Kriging

Answer 29

The mean is estimated as well as the parameters in the variogram

Question 30

Define Universal Kriging

Answer 30

The mean is a function of some covariates, usually but not exclusively, spatial co-ordinates.

Question 31

Different weights to use when fitting the theoretical variogram to the sample variogram by least squares?

Answer 31

1. number of pairs in each bin - more data so trust more thus larger weighting than elsewhere 2. the theoretical variogram - where variogram is higher, trust it more(?) 3. equal weights

Question 32

Outline the steps in the Method of Moments to fit a variogram?

Answer 32

Calculate the sample variogram Choose a shape for the variogram Fit that variogram to the sample variogram by weighted least squares

Question 33

What is Hawkin's and Cressie used for?

Answer 33

Hawkins and Cressie is an alternative to estimating the sample variogram.

Question 34

Name some loss functions used to get a point estimate from the posterior distribution.

Answer 34

squared loss = mean of the posterior; absolute loss = median; (0, 1) loss = mode (also known as maximum a posteriori (MAP) estimates)

Question 35

Name different types of priors

Answer 35

``` Subjective Bayes Objective Bayes Conjugate Priors Non-Informative Priors Informative Priors (MCMC methods) ```

Question 36

2 varieties of MCMC methods

Answer 36

Gibbs Sampler | Metropolis-Hastings

Question 37

Describe a conjugate prior

Answer 37

Conjugate prior is one such that the formula for the posterior and the prior are the same

Question 38

What type of prior is mostly used in Bayesian Inference?

Answer 38

Improper priors / Non-informative priors

Question 39

Different methods of obtaining a prior distribution of the length scale (delta) in Bayesian Inference.

Answer 39

Maximise the posterior (MAP) MCMC Approximate the posterior and sample from that Discretise the prior on δ

Question 40

Discretising the prior on δ uses what method?

Answer 40

Monte Carlo

Question 41

Different validation methods for fitting a GP

Answer 41

Leave one out Leave N out If a completely independent data set, hold some back and use to check (Indivudual Prediction Errors)

Question 42

For Leave one out method, how many points would you expect to see outside +/- 2 standard deviations?

Answer 42

1 in 20

Question 43

Name some of the 'inputs' for a simulator/computer model?

Answer 43

Parameters ICs BCs

Question 44

What are reasons for the uncertainty in simulators/computer models?

Answer 44

STRUCTURAL UNCERTAINTY; - Uncertainty in the underlying science (Don't know the world perfectly, hence equations not perfect) - Uncertainty in the solution of the equations (the discretisation adds additional uncertainty etc) UNCERTAINTY IN THE INPUTS

Question 45

Define an emulator

Answer 45

A Gaussian process to model the simulator output as a function of its input

Question 46

What is Sensitivity Analysis? (Emulators)

Answer 46

How sensitive is the simulator output to a change in an input (or combination of inputs)

Question 47

What is Uncertainty Analysis? (Emulators)

Answer 47

If we are uncertain about the simulator inputs what does that say about our uncertainty on the outputs

Question 48

Name two designs for a set of simulator runs to span input space

Answer 48

Optimised Latin Hypercubes and quasi-Monte Carlo sequences(Sobol)

Question 49

Two ways of assessing how well spread a Latin Hypercube is?

Answer 49

``` Maximin - maximise the minimum distance between points Orthogonal designs (e.g. good coverage on x1 + x2 ) ```

Question 50

Idea behind discrepancy (theory) in space?

Answer 50

Distributing points in some space such that they are evenly distributed with respect to some (mostly geometrically defined) subsets. The discrepancy (irregularity) measures how far a given distribution deviates from an ideal one.

Question 51

Steps in Building an Emulator

Answer 51

Specify the Gaussian process model (mean and covariance function) Select the prior distributions for the GP hyperparameters Choose a design for training and validation Run the ensemble of model runs Fit the emulator to the simulator runs Validate and re-fit if needed

Question 52

Why can't time series data be modelled like spatial data?

Answer 52

Time has one direction Time series is considered as discrete as it is often collected at regular intervals (Trends more common in time series - need to consider more) Time series are used to extrapolate, whereas spatial data is normally used for interpolation

Question 53

2 ways of dealing with time series seasonality?

Answer 53

Seasonal anomalies | Seasonal differences

Question 54

How to describe data that has second order stationarity (weak) and Gaussian?

Answer 54

Strictly Stationary | Strict stationarity implies weak/second-order stationarity and the converse is true for Gaussian processes only.

Question 55

Which type of stationarity is not (equivalently) defined in time series data?

Answer 55

Intrinsic stationarity

Question 56

Can time series give negative correlations?

Answer 56

Yes

Question 57

Assumption on the variance for Auto correlation functions?

Answer 57

Variance = 1

Question 58

Name of the time series equivalent of the Bochner Theorem

Answer 58

The Wiener-Khintchine Theorem

Question 59

The Wiener-Khintchine Theorem

Answer 59

The Fourier transform of a valid covariance (correlation) function is a density function (and vice versa) Allows you to talk about time series in a Fourier Space

Question 60

Relationship of spectrum and ACF?

Answer 60

The fourier transform of the ACF is called the Spectral Density Function (spectrum) NB the fourier transform of the spectrum is also the ACF

Question 61

Name a condition for calculating the spectrum

Answer 61

mean = 0

Question 62

How to calculate the Cross Spectrum?

Answer 62

Take the fourier transform of the cross-covariance between 2 time series x_t and y_t

Question 63

Describe the form of the Cross Spectrum

Answer 63

``` Complex number S_xy = c_xy − iq_xy c_xy is the co-spectrum q_xy is the quad-spectrum S_yx is the complex conjugate of S_xy ```

Question 64

Define coherency between two time series x and y

Answer 64

Coherency = correlation between the x and y at a particular frequency

Question 65

How to represent the Cross spectrum in terms of Amplitude and Phase

Answer 65

Can because of complex number Amplitude: sqrt[(c_xy)^2 + (q_xy)^2] Phase: arctan(c_xy/q_xy)

Question 66

Is kriging model-free?

Answer 66

Yes

Question 67

Is spectral analysis non-parametric?

Answer 67

Yes | It is model-free

Question 68

What type of equations are the MA and AR processes?

Answer 68

Difference equations

Question 69

What is ε_t representing in the MA and AR processes?

Answer 69

εt is white noise. εt is i.i.d (independently and identically distributed) from a normal distribution with mean zero and with variance σ^2_w

Question 70

How to find the auto-correlation function, ρ(h), of a MA or AR process?

Answer 70

Auto-covariance function / variance of the process h is the lag ρ(h=0) = 1

Question 71

MA(1) process

Answer 71

``` x_t = ε_t + β1*ε_{t-1} x_t = (1 + β1*B)ε_t ```

Question 72

AR(1) process

Answer 72

x_t = α*x_{t-1} + ε_t

Question 73

Condition for stationary AR(1) process

Answer 73

|α| < 1 ~ stationary process α = 1 ~ random walk |α| > 1 ~ explosive process

Question 74

Relationship between AR and MA models

Answer 74

An AR(q) model is actually an MA model with infinite order. An MA model is an infinite order AR model. (Can be shown for q=1 and for any q i.e. all AR processes)

Question 75

Another term for white noise variance

Answer 75

Innovation variance

Question 76

Contextually describe the PACF?

Answer 76

PACF: what correlation do I have in the next lag, that isn't explained by all the previous lags

Question 77

How to write an ARMA process in its causal and invertible form?

Answer 77

Causal form, define xt in terms of εt (put everything on the MA side of the ARMA model) Invertible form, define εt in terms of xt (put everything of the AR side of the ARMA model)

Question 78

When is an ARMA process causal?

Answer 78

Let θ(z) and Φ(z) be the AR and MA polynomials with B (the backward shift operator) replaced with a complex number z. An ARMA process is causal iff Φ(z) =/= 0 for |z| <= 1

Question 79

When is an ARMA process invertible?

Answer 79

Let θ(z) and Φ(z) be the AR and MA polynomials with B (the backward shift operator) replaced with a complex number z. An ARMA process is invertible iff θ(z) is =/= 0 for |z| <= 1

Question 80

Are MA processes unique?

Answer 80

No

Question 81

Are AR(p) models linear or non-linear?

Answer 81

Linear

Question 82

How to choose order of ARMA model?

Answer 82

Best AIC or BIC values Akaike Information Critera (AIC) Bayesian Information Critera (BIC)

Question 83

What time series model can be used to capture seasonality?

Answer 83

ARIMA | DLM

Question 84

Indication of a good fitting time series model? (AR, MA, ARMA, ARIMA)

Answer 84

Residuals should be normally distributed | The residuals should be uncorrelated (can tell by ACF, or spectrum)

Question 85

How to forecast using time series models? (AR, MA, ARMA)

Answer 85

AR can be done differently because linear; One step ahead prediction error, by minimising the mean squared prediction error - substituting in the best linear predictor. ARMA and MA; The Durbin-Levinson Algorithm

Question 86

What part of the DLM is not observed?

Answer 86

The state space | What we're most interested in...

Question 87

In which model can the error variance change with time?

Answer 87

DLM | Dynamic Linear Model

Question 88

What are the names of the 2 equations in a DLM?

Answer 88

State equation | Observation equation

Question 89

Notation for the state and observation regression matrices in their respective equations?

Answer 89

G_t (Φ_t) in state equation | F_t (A_t) in observation equation (F^T)

Question 90

Name and describe the quadruple of matrices defining a DLM

Answer 90

REGRESSION MATRICES F_t (A_t) ~ for the observation equation G_t (Φ_t) ~ for the state equation VARIANCE MATRICES OF THE NORMAL DISTRIBUTION OF THE ERROR TERM V_t ~ for the observable equation W_t ~ for the state equation

Question 91

If the ‘regression’ matrices are constant in time the DLM is known as what? And if the variance matrices are also constant?

Answer 91

Time Series DLM (TSDLM) | Constant DLM

Question 92

Describe the properties of a univariate DLM

Answer 92

``` A Univariate DLM has y_t (Y_t) and v_t univariate. Note x_t (θ_t) can still be a vector in a univariate DLM. ```

Question 93

Difference between Forecasting, Filtering and Smoothing

Answer 93

We are trying to estimate the properties of the state (x_t) from the data (y_s) If t > s this is forecasting (using only the past) If t = s this is filtering (using the past and present) If t < s this is smoothing (using past, present and future)

Question 94

What ways can you fit a DLM to the data (i.e. estimate the parameters)?

Answer 94

Numerically MLE Bayes Apart from some special cases MCMC Gibbs sampler (Usually use this)

Question 95

What is the Kalman Filter?

Answer 95

A way to forecast data

Question 96

Define data assimilation and name 2 ways in which this can be done.

Answer 96

Combine data with output from a numerical model to make a better forecast 1) Kalman filter 2) Variational Methods

Question 97

What is a particle filter

Answer 97

Run an ensemble of models that are spread enough to | allow us to calculate P_ft

Question 98

Two approaches to dealing with non-stationary spatial fields (non-stationary covariance function).

Answer 98

1. CHANGE SPACE: Warp space so that our conventional methods work 2. CHANGE METHODS: Explicitly use a GP model that includes the non-stationarity

Question 99

General form of GP

Answer 99

y(x) = µ(x) + σ(x) + ε(x) ``` • y(x) - our output • µ(x) - deterministic mean function • σ(x) - zero mean GP • ε(x) - nugget to cope with measurement error and small scale variability ```

Question 100

Give names to the 'spaces' when warping space to deal with non-stationary spatial fields.

Answer 100

* Geographic space is G-space | * Deformed space is D-space (D=f(G))

Question 101

What is an identifiable model?

Answer 101

A model where you can estimate all the parameters

Question 102

What are the three different approaches to modifying the methods/theory when accounting for non-stationarity in the covariance matrix for spatial fields?

Answer 102

1. Use a non-stationary covariance function (scale/marginalise) 2. Use the process convolution definition of GPs 3. Reformulate the GP as the solution to a stochastic partial differentiable equation

Question 103

The kernel function is equivalent to what?

Answer 103

The covariance function

Question 104

What does SPDE stand for and what is it?

Answer 104

Stochastic Partial Differential Equation (NB, not deterministic) These are Partial Differential Equations driven by white noise.

Question 105

What SPDEs have a solutions that are Gaussian Processes?

Answer 105

LINEAR SPDEs | The solution of all linear SPDEs are Gaussian processes.

Question 106

What is the precision matrix?

Answer 106

The inverse of the covariance matrix

Question 107

What useful property does the precision matrix have?

Answer 107

It is sparse So can use sparse-matrix methods Which are fast calculations

Question 108

What does INLA stand for and what is it?

Answer 108

Integrated Nested Laplace Approximation A procedure that uses the Laplace approximation to approximate the posterior for a set of Gaussian models (including GMRF)

Question 109

What is INLA an alternative to?

Answer 109

MCMC for Bayesian inference

Question 110

In Spatio-temporal modelling, what do S_s and S_t represent?

Answer 110

S_s ~ spatial variance matrix (qxq) | S_t ~ temporal variance matrix (pxp)

Question 111

Name a method, widely used in climate and weather forecasting, for spatio-temporal modelling.

Answer 111

Empirical Orthogonal Functions (EOFs) | PCA

Question 112

What does separability imply about spatio-temporal data?

Answer 112

Separability implies that - the spatial correlation structure does not change with time and that - the time structure does not change with space. The covariance function in space and time can be separated into a spatial part and a temporal part.

Question 113

Name a model to use to model spatio-temporal data that is non-seperable?

Answer 113

Generalised DLM | Coregionalisation?

Question 114

Which is faster? INLA or MCMC?

Answer 114

INLA