W11 Validating Simulation Models

Question 1

Verification

Answer 1

does the code ask what the specification asks?

Question 2

Structural Validation?

Answer 2

Does model correctly represent problem space?

Question 3

Input/Output Validation

Answer 3

Does simulation correctly represent an instance of hte problem.
Calibration:
adjust input values to get valid output values

Question 4

Simulation Data (3 types)

Answer 4

Input Data
Process Data
Output (Result) Data

Question 5

Input Data

Answer 5

collected or inferred from empirical status quo

Question 6

Process Data

Answer 6

generated during simulation process

provides insights into model that are not empirically available or relevant for simulation purpose

Question 7

Output (Result) Data

Answer 7

Indicators calculated for validation or what-if analysis, matching empirical indicators

Question 8

Input Data:

kinds of data?

Answer 8

direct observations
->transparent systems

indirect inference
->empirical process and result indicators, e.g. customer choice

Scenarios:
stochastic
best/worst case
discrete

Question 9

Railway Ticket as Input.

What data?

Answer 9

directly observable:
supply: products capacity, price categories, availabilities

demand: historical sales

indirect inference:
customer loyalty
reference prices
wtp

Question 10

Input Data on the Future

Answer 10

1 generate insights about empirical data

2 simulate future scenarios by forecasting from 1

Question 11

Process Data Types

Answer 11

empirically-available

• forecasts
• plans
• documented events

simulation-exclusive process data

• decisions
• learned experience
• communications

Question 12

Result Data Types

Answer 12

empirically available

• system reports
• transaction data

simulation exclusive

• long term developments
• what if developments
• internal states

Question 13

Usage of Results Data

Answer 13

output validation (compare results to empirical data)

sensitivity and meta modeling: analyze relationship in the black box
data farming: simulations generate artificial transaction data

Question 14

Data Farming

Answer 14

simulation models run multiple times to provide insights into different consequences of different options

• generate much data
• success dependent on validity
• simple: monte carlo
• mechanik: discrete
  emergent: agent-based

Question 15

Monte Carlo Simulations

Answer 15

Problem: too little empirical data
Idea: create data based on fitting distributions
Change: meaningful conclusions from enriched sata
Risk: tainted by assumptions about distribution

Question 16

Verification

Answer 16

50/50

not your own code

Question 17

Verification: Unit Test Frameworks

Answer 17

Systematic test case design, implementation and analysis

• tested data and methods
• expected results

Question 18

Structural Validation

Answer 18

does model correctly represent the problem space?

• systematically explicate model components and relationships
• expert workthrough

Q: are all relevant concepts and structures included?
Q: is model structure consistent with relevant knowledge of system?

Question 19

Input Validation

Answer 19

Match probability distributions
Match expected future scenarios

Q: matching empirical observations
Q: are parameter values consistent with descriptive numerical knowledge?

Question 20

Output Validation

Answer 20

behavioral validation

does simulation output match empirical observations?

Question 21

Error metric handling

Answer 21

how much error acceptable?
how much confidence is enough?
sufficient fit when given?

Question 22

Calibration

Answer 22

adjusting inputs for better outputs

Question 23

Calibration Approaches

Answer 23

rigorous
pragmatic
many parameters, realistic
few parameters, abstract