11 - Validating simulation models

Question 1

Motivation: Why validate simulations?

Answer 1

• when not rigorously calibrated and validated, simulations are neither a reliable research method nor a reliable tool for practical decision support

Question 2

Definition: Verification, Validation, Calibration
(Steps, that subsume validation)

Verification

Answer 2

Does the code do what the specification asks?

- code could include faults

Question 3

Definition: Verification, Validation, Calibration
(Steps, that subsume validation)

Structural Validation

Answer 3

Does the model correctly represent the problem space?

- are relevant relationships included in the model etc?

Question 4

Definition: Verification, Validation, Calibration
(Steps, that subsume validation)

Input/Output validation

Answer 4

Does the simulation correctly represent the problem space?

• some kind of description of the status quo based on data
• parametrise the shape of the distribution

Calibration: adjusting input values to get valid output values
-> danger: overfitting the model when calibrating it

-> agent-based models are subject to over parametrization -> customer learning -> we can make assumptions but we don’t know the parameter for sure (no input parameters)

Question 5

Simulation Data

Different layers of data

Answer 5

Input data
Process data
Output (result) data

Question 6

Simulation Data

Different layers of data

Input data

Answer 6

• collected of inferred from empirical status quo

- e.g. customer arrival rate, range of products a company offers

Question 7

Simulation Data

Different layers of data

Process data

Answer 7

• generated during the simulation process, provides insights into the simulation model that are not empirically available or relevant for the simulation purpose
• process or event logs from the real world can be compared

Question 8

Simulation Data

Different layers of data

Output (Results) data

Answer 8

• indicators calculated for validation or what-if analysis, matching empirical indicators
• in the status quo these are the indicators we are interested in -> what happens when we change them? Outcome?

Question 9

Simulation data

Input data

Answer 9

Assumption:
- Structure of relevant simulation components and parameters has been determined

Direct observation:

• for transparent systems
• example: machine run times
• or: what is the actual outline of the shop

Indirect inference:

• based on empirical process and result indicators
• example: customer choice
• > we know what happened but not why, so we apply data analysis to find a model how things work in the real world

Question 10

Simulation data

Input data: scenarios

Answer 10

Stochastic scenarios
Worst- and best case scenarios
Qualitatively discrete scenarios

Question 11

Simulation data

Input data: scenarios

Stochastic scenarios

Answer 11

• follow empirical distributions
• stochastic input scenarios lead to stochastic process and result data
• > when we are uncertain about data -> consider stochastic scenarios

Question 12

Simulation data

Input data: scenarios

Worst- and best-case scenarios

Answer 12

• model extreme cases
• results indicate the modeled systems’ robustness
• > robustness = does it behave the same in each of these cases?

Question 13

Simulation data

Input data: scenarios

Qualitatively discrete scenarios

Answer 13

• model discrete alternative cases

- test robustness and the necessity of individualized strategies

Question 14

Simulation data

Example: railway ticket as simulation input

Answer 14

Directly observable:

• supply: products, capacity, price categories, availabilities
• demand: historical sales

Indirect inference (also: estimation):

• customer loyalty (e.g. we have the name on the ticket -> we can analyse how often they bought in the past)
• reference prices
• willingness to pay
• > can use this to calibrate customers in a model

Question 15

Simulation data

Input data on the future

Answer 15

1. Insights:
   - analyzing empirical data to parametrize the simulation input data is based on insights, the pre-condition for predictive analytics
2. Forecast:
   - to simulate future scenarios, the future values of input data have to be forecasted

Question 16

Simulation data

Process data

Answer 16

Simulation systems are fully transparent - all data that is generated in the process of the simulation can be observed

We may look at: empirically-available process data

• forecasts
• plans
• documented events

And we can compare it to: simulation-exclusive process data (= generate process data that wouldn’t have been availble in the real world):

• decisions
• learned experiences
• communication

Question 17

Simulation data

Result data

Answer 17

Usually empirically-available

• system reports
• transaction data

Simulation-exclusive

• long-term developments
• what-if developments
• internal agent- and system-states: e.g. customer satisfaction, emergent strategies

Question 18

Simulation dara

Result data

Usage

Answer 18

• Output validation: compares result data to empirical data
• sensitivity and meta-modeling: analyse relationship in the “black box”
• data farming: simulations generate artificial transaction data -> data of the real world is not sufficient so we need more data

Question 19

Simulation data

Data farming

Answer 19

= simulation models run thousands of times can provide insights into the different consequences of different options

• generates as much reproducible data as desired (amount is only limited to time and storage space)
• success depends on validity - which is difficult to determine for human, social, cultural and behavioral modeling

Question 20

Simulation data

Data farming

Which types of data farming are there?

Answer 20

Simple: Monte Carlo
- Create data from known distributions

Mechanic: Discrete event-based white box

• create data given varying scenarios
• look at everything that happens in the real world -> input data

Emergent: Agent-based black box

• create data based on assumptions and theories
• we don’t know what’s actually going on in the empirical agents

Question 21

Simulation data

Data farming with Monte-Carlo Simulations

Problem:
Idea:
Chance:
Risk:
Example:

Answer 21

Problem:
- empirical data set is not large enough to allow for significant statements about variable relationships

Idea:
- create additional data based on distributions fitted to the empirical sample (fill in potential gaps)

Chance:
- draw more meaningful conclusions from the enriched data set

Risk:
- enriched data set is “tainted” by a priori assumptions about distributions (gaps are filled based on assumptions)

Example:
- survey responses on the influence of consultative committees

Question 22

Verification

Answer 22

aims to ensure that the code does what the specification asks - identify and eliminate “bugs”

Problem: when to stop testing?

• testing takes a creative and destructive mind
• avoid testing your own code
• test cases should be formulated by subject matter experts (don’t use the same people to develop and test -> won’t notice mistakes)
• test cases only prove the absence of those errors that they were designed to test

Question 23

Structural validation

Answer 23

Does the model correctly represent the problem space?

• systematically explicate model components and relationships
• expert walkthrough with mode stakeholders

Questions:

• Are all concepts and structures relevant to the problem included?
• is the model structure consistent with relevant knowledge of the system?

Question 24

Structural validation

ODD protocol

Answer 24

• used to describe individual-based models, agent based models, simulation models

Overview:

1. Purpose
2. Entities, state variables, and scales
3. Process overview and scheduling

Design:
4. Design Concepts (Basic principles, emergence, adaptation, objectives, learning, prediction …)

Details:

5. Intitialization
6. Input data
7. Submodels

Question 25

Input data validation

Input validation

Answer 25

• sometimes defined as an aspect of structural validation definition
• design input probability distributions that match the empirical observations
• design input parameter values that match expected future scenarios

Questions:

• Do inputs correspond to empirical observations?
• Are the parameter values consistent with descriptive numerical knowledge?
• > parametrizing can show that there’s a lack of information
• > e.g. building a simulation to test different settings for the shop floor -> you find out that manufacturer doesn’t know the machine runt time
• > if input variables cannot be validated, output cannot be validated either

Question 26

Output validation

Answer 26

• Does the simulation output match empirical observations?
• also called behavioral validation
• > run the simulation model and generate output data -> you have to wait for the last minute to do output validation
• > compare if the simulation does what we observe in the real world
• > e.g. model the queue precision correctly

Question 27

Output validation

Questions of output validation

Answer 27

What is the average distance between empirical observations and simulation results?

• percentage: MAPE
• absolute: RMSE
• bias
• > How much error is acceptable?

Do confidence intervals overlap?
-> How much confidence is enough?

Does a meta-model fitting the empirical observations also fit the simulation? (E.g. a regression model)

• for this, empirical input and output information needs to be available
• > What fit suffices for the empirical model, anyway?
• > How closely does the regression model match?

Question 28

Output validation

Cross-validation

Answer 28

• the data set is split into training, validation and test set
• to calibrate the simulation, you use the training and validation set
• then you use the test set for test validity