Sargent (validation and verification)

Question 1

Why do validation:

Answer 1

In order to ensure that the model works for the intended purpose.
In problem structuring the purpose will often be to achieve clarity and to map interest and conflicting actions.

Question 2

What is model verification according to Sargent (2013)?

Answer 2

Ensuring the computer programming and its implementation of the conceptual model are done correctly

Question 3

What is model verification according to Sargent (2013)?

Answer 3

Ensuring the computer programming and its implementation of the conceptual model are done correctly

Question 4

What is model validation according to Sargent (2013)?

Answer 4

Ensuring that the model serves it purpose

that a model works well and gives accurate results within its intended use.

Question 5

What is model credibility according to Sargent (2013)?

Answer 5

Is concerned with developing confidence in the user, which is required to use the model and the derived information from the model

Question 6

When is a model considered valid for experimental condition according to Sargent (2013)?

Answer 6

if the model’s accuracy falls within the desired range for its intended purpose, then it is considered acceptable.

Question 7

Full validated model vs sufficient accuracy:

Answer 7

Often, we will not fully validate a model –> too costly.

Validate it to some extend. It will be sufficient accurate to the purpose of the model.

Doing more validation creating more confidence in the model also result in higher cost
The value of a model increases with the confidence in a model (validity) but at a decreasing rate

Question 8

Three decision making approaches for deciding on simulation model validity (Sargent 2013)

Answer 8

All approached require the model development team to conduct verification and validation as part of the model development process

1) The model development team decide themselves if the model is valid
→ This is based on result of test and evaluations

2) The users of the simulation model is determining the validity of the model.
→ Better than above
→ Here the users are heavily part of the model development process when doing validation and verification

3) Use a third party to decide on the simulation models validity
→ Called an independent verification and validation (IV&V)
→ Often used when the model developed are very large
→ The IV&V team can conduct the validation during the development of model or after the model is developed.

• When done during the development team will not move on to the next stage before the IV&V team has evaluated it to satisfy the requirement for the current stage
• When done after the development of the model it is often extremely costly and time consuming

Question 9

Explain the simplified version of the model development process (Sargent 1981):

Answer 9

Problem entity is the system or phenomena to be modelled

Conceptual model is the mathematical/logical/graphical representation of the problem
- Developed through analysis and modelling phase

Computerized model is the conceptual model implemented on a computer
- Developed through a computer programming and implementation phase

inferences about the problem entity are obtained by
conducting computer experiments on the computerized model in the experimentation phase.

Conceptual model validation is determining that theories and assumption underlying the conceptual model are correct and the model representation of the entity is reasonable for the intended purpose

Computerized model verification: Assuring that the computer programming and implementation of the conceptual model are correct

Operational validation determines that the models accuracy is within the range of accuracy for the models intended purpose

Data validity ensures that the data necessary for model building, model evaluation and test are adequate and correct

The process is iterative and every time any changes are made the process start over again

Question 10

Animation as validation technic (Sargent 2013)

Answer 10

The model’s operational behavior is displayed graphically as the model moved trough time

For instant the movement of parts in a factory
Often not all behaviors being observed since it is often done in a short time interval

Question 11

Comparison to other models as validation technic (Sargent 2013)

Answer 11

Various results of the simulation model is being validated by comparing it to results of other (valid) models

Question 12

Data relationship correctness as validation technic (Sargent 2013)

Answer 12

Data relationship correctness validation ensures accurate and consistent connections between data elements in a model or system.

Question 13

Degenerate tests as validation technic (Sargent 2013)

Answer 13

A test of the models behavior is done by appropriate selection of values of input and internal parameters.
For instant does the average number in the queue continue to increase over time, when the arrival rate is larger than the service rate ?

Question 14

Event validity as validation technic (Sargent 2013)

Answer 14

The “events” of occurrences of the simulation model are compared to those of the real system to determine whether they are similar
For instant: compared the number of fires in the fire department simulation to the actual number of fires

Question 15

Extreme condition test as validation technic (Sargent 2013)

Answer 15

The model structure and outputs should be plausible for any extreme and unlikely combination of levels of factors in the system

Question 16

Face validity as validation technic (Sargent 2013)

Answer 16

Face validity is a subjective assessment of whether a model measures what it claims to measure based on its content and relevance.

For example is the logic in the conceptual model correct and are the models input-output relationship reasonable?

Question 17

Historical data validation as validation technic (Sargent 2013)

Answer 17

If historical data exist, part of the data is used for build the model and remaining data are used to test whether the model behaves as the system does

Question 18

Internal validity as validation technic (Sargent 2013)

Answer 18

Several replications (runs) of a stochastic model are made to determine the amount of (internal) stochastic variability in the model
If there are large variation in the results it might make the model and the results questionable

Question 19

Operational graphics as validation technic (Sargent 2013)

Answer 19

Performance measure are shown graphically while the model is running through time
The performance measure could be number un queue or percentage og servers which are busy

Question 20

Parameter variability-sensitivity analysis as validation technic (Sargent 2013)

Answer 20

Changing values and input or internal parameter of the model to determine the effect at the model´s behavior or output.
The same relationship should occur in the model as in the real system.

Question 21

Predictive validation as validation technic (Sargent 2013)

Answer 21

The model predicts (forecast) the system´s behavior. A comparison is made between these to see if they are the same.

Question 22

Conceptual model validation according to Sargent (2013)

Answer 22

1) The theories and assumptions for the conceptual model are correct → Mathematical analysis and statistical methods on problem entity data.

2) The model’s representation of the problem entity and the model’s structure is reasonable for the intended purpose of the model → Has the appropriate detail and aggregate level been used?

Primary validation techniques are
- Face validation, structured walkthrough and trace

Question 23

Computerized model verification according to Sargent (2013)

Answer 23

Ensures that the computer programming and implementation of the conceptual model are correct

Improving the verification can be done by using a simulation language

Primary technics are Structured walkthroughs and trace

Question 24

Approached for testing simulation software (Sargent 2013)(Fairly 1976)

Answer 24

Static testing: the computer program is analysed to see if it is correct
- Structured walkthrough, correctness proofs and examining the structure properties of the program
Dynamic testing: The computer program is executed under different conditions and the values obtained are used to decide if the program and its implementation are correct
- Trace, input-output relationships, data relationship correctness

Question 25

Operational validity according to Sargent (2013)

Answer 25

Operational validation is determining if the simulation model’s output behavior has the accuracy required for the intended purpose of the model Major factor for this validity is whether the problem entity (the system) is observable - Observable means possible to collect data on the operational behavior of the entity

Question 26

Explore model behavior (operational validation Sargent 2013))

Answer 26

Can be done either qualitatively or quantitatively In qualitatively analysis the direction of the output behavior are examined In quantitatively analysis both the direction and the precise magnitude of the output behavior is examined Model exploration can be done using many of the technics we have had about, but parameter variability -sensitivity analysis is one to be used.

Question 27

Comparisons of output behaviors (operational validation Sargent 2013)

Answer 27

Three basic approaches in comparing the simulation model output behavior to either the system or another model (output behavior) - Hypothesis test (Objective decision) - Confidence intervals (Objective) - Graphs (Subjective)

Question 28

Hypothesis test (Operational validation Sargent 2013)

Answer 28

Objective Can be used in comparison of means, variance, distributions and time series of the output variables of a model to determine if the simulation model’s output behavior has a satisfactory range of accuracy First hypothesis should be stated in order for them to be tested Two possible errors when using hypothesis test - Type I: rejecting the validity of a valid model – model builder’s risk - Type II: accepting the validity of a invalid model – model user’s risk

Question 29

Confidence intervals (Operational validation Sargent 2013)

Answer 29

Objective Can be obtained for the differences between means, variances and distributions for variables The confidence intervals can be used as the model range of accuracy for model validation

Question 30

Graphical comparisons of data (Operational validation Sargent 2013)

Answer 30

Subjective decision is made on each graph whether it is within the accuracy is within the acceptable range Data of the simulation model and system output variables are graphed for different experimental conditions to decide if the models output behavior has sufficient accuracy Three types of graph are used - Histograms - Boxplot - Behavior graphs (Scatter plot to show relationship) Here it is also possible to make Type I and type II errors like in hypothesis test.

Question 31

Recommended procedure for validation (Sargent 2013)

Answer 31

1) Make agreement between model development team, model sponsors and users about the decision making approach and specific validation technics to be used 2) Specify the acceptable range of accuracy – should be done before development of model or early in the development 3) Test assumptions and theories underlying the model 4) In every iteration perform face validity on conceptual model 5) Every model iteration explore the simulation model behavior using computerized model 6) At least in last model iteration, make comparisons if possible between model and system 7) Prepare verification documentation 8) If the model is to be used over time develop a schedule for periodic review of the model validity

Question 32

Objective vs subjective use of validation technics (Sargent 2013)

Answer 32

Objective: using mathematical procedure or statistical test - For instant hypothesis test or confidence intervals Subjective: When a subjective decision is made whether the validity is acceptable - For instant using graphs which rely on subjective conclusion and evaluation

Question 33

Validation characterized (managerial perspective)

Answer 33

The problem validation from a managerial perspective - characterized by the following properties: 1. The processes of modeling and validation cannot be done separately 2. Consider model validation as a set of interconnected decisions to be made 3. To ensure the usefulness dimension of validity, an active participation of the main actors in the modeling-validating process and the consideration of the problem context in which this is carries out are emphasized.

Question 34

Model assessors

Answer 34

A group independent of both model builders and model users → Necessitates a more explicit integration of model validation and documentation into the process of OR studies.

Question 35

What does model validation involve?

Answer 35

Involves usefulness, usability, and cost considerations and representativeness within the context of a modeler-user interface

Question 36

Purpose of the modeling-validation process

Answer 36

Aims to link the processes of the model building and the model validation into a single process

Question 37

Why is it important to identify the stakeholder (users of the model) in the model-validating process?

Answer 37

It is a decision-system that involves human factors  so the model involves value. In order to determine the favor values for the model the stakeholders should be determined early in the process. The success or failure of a model depends very much on the attitude and behavior of stakeholders (the model’s clients)

Question 38

The types of model objective

Answer 38

1)Prediction: Predict future values (or behavior) of some variables without necessarily desiring to understand the internal functioning of the system. 2)Comprehension: Better understand the problem situation without necessarily desiring to predict its future behavior.

Question 39

Explain the importance-certainty graph

Answer 39

Consider the degree of importance and certainty of each relevant assumption. Useful in providing a deeper insight into the problem situation, → forces model builders and decision-makers to clarify and compare the relevant assumptions with respect to each other → The model builder is then in a better position to judge where to put effort

Question 40

What is meant by model confidence?

Answer 40

The user’s total attitude toward the model, and of the willingness to employ its results in making decisions. Decision makers need to have confidence in the model and its output before using is for making decisions

Question 41

Which costs are associated with a model:

Answer 41

Two types: 1) Cost of developing the model and 2) Cost of using and implementing the model These two costs are usually in conflict their sum is a convex function of the level of validity. The appropriate level of validity is the one which minimizes the sum of the associated costs

Question 42

Turing test:

Answer 42

Individuals who are knowledgeable about the operations of the system being modelled are asked whether they can discriminate between system and model outputs.

Question 43

Block box validation

Answer 43

In black-box validation, the model’s internal construction is treated as being unknown The validation focuses on the predictive power of the model The validation is based on a comparison of two sets of observations: - one set from the model, - the other from a reference system (the real system). Statistical tests can be applied in order to assess the similarity between the two sets of observations

Question 44

White/open box validation

Answer 44

In open-box validation, the detailed internal structure of the model is compared with that of its reference system (the real system)