Sargent (validation and verification) Flashcards
Why do validation:
In order to ensure that the model works for the intended purpose.
In problem structuring the purpose will often be to achieve clarity and to map interest and conflicting actions.
What is model verification according to Sargent (2013)?
Ensuring the computer programming and its implementation of the conceptual model are done correctly
What is model verification according to Sargent (2013)?
Ensuring the computer programming and its implementation of the conceptual model are done correctly
What is model validation according to Sargent (2013)?
Ensuring that the model serves it purpose
that a model works well and gives accurate results within its intended use.
What is model credibility according to Sargent (2013)?
Is concerned with developing confidence in the user, which is required to use the model and the derived information from the model
When is a model considered valid for experimental condition according to Sargent (2013)?
if the model’s accuracy falls within the desired range for its intended purpose, then it is considered acceptable.
Full validated model vs sufficient accuracy:
Often, we will not fully validate a model –> too costly.
Validate it to some extend. It will be sufficient accurate to the purpose of the model.
Doing more validation creating more confidence in the model also result in higher cost
The value of a model increases with the confidence in a model (validity) but at a decreasing rate
Three decision making approaches for deciding on simulation model validity (Sargent 2013)
All approached require the model development team to conduct verification and validation as part of the model development process
1) The model development team decide themselves if the model is valid
→ This is based on result of test and evaluations
2) The users of the simulation model is determining the validity of the model.
→ Better than above
→ Here the users are heavily part of the model development process when doing validation and verification
3) Use a third party to decide on the simulation models validity
→ Called an independent verification and validation (IV&V)
→ Often used when the model developed are very large
→ The IV&V team can conduct the validation during the development of model or after the model is developed.
- When done during the development team will not move on to the next stage before the IV&V team has evaluated it to satisfy the requirement for the current stage
- When done after the development of the model it is often extremely costly and time consuming
Explain the simplified version of the model development process (Sargent 1981):
Problem entity is the system or phenomena to be modelled
Conceptual model is the mathematical/logical/graphical representation of the problem
- Developed through analysis and modelling phase
Computerized model is the conceptual model implemented on a computer
- Developed through a computer programming and implementation phase
inferences about the problem entity are obtained by
conducting computer experiments on the computerized model in the experimentation phase.
Conceptual model validation is determining that theories and assumption underlying the conceptual model are correct and the model representation of the entity is reasonable for the intended purpose
Computerized model verification: Assuring that the computer programming and implementation of the conceptual model are correct
Operational validation determines that the models accuracy is within the range of accuracy for the models intended purpose
Data validity ensures that the data necessary for model building, model evaluation and test are adequate and correct
The process is iterative and every time any changes are made the process start over again
Animation as validation technic (Sargent 2013)
The model’s operational behavior is displayed graphically as the model moved trough time
For instant the movement of parts in a factory
Often not all behaviors being observed since it is often done in a short time interval
Comparison to other models as validation technic (Sargent 2013)
Various results of the simulation model is being validated by comparing it to results of other (valid) models
Data relationship correctness as validation technic (Sargent 2013)
Data relationship correctness validation ensures accurate and consistent connections between data elements in a model or system.
Degenerate tests as validation technic (Sargent 2013)
A test of the models behavior is done by appropriate selection of values of input and internal parameters.
For instant does the average number in the queue continue to increase over time, when the arrival rate is larger than the service rate ?
Event validity as validation technic (Sargent 2013)
The “events” of occurrences of the simulation model are compared to those of the real system to determine whether they are similar
For instant: compared the number of fires in the fire department simulation to the actual number of fires
Extreme condition test as validation technic (Sargent 2013)
The model structure and outputs should be plausible for any extreme and unlikely combination of levels of factors in the system
Face validity as validation technic (Sargent 2013)
Face validity is a subjective assessment of whether a model measures what it claims to measure based on its content and relevance.
For example is the logic in the conceptual model correct and are the models input-output relationship reasonable?
Historical data validation as validation technic (Sargent 2013)
If historical data exist, part of the data is used for build the model and remaining data are used to test whether the model behaves as the system does
Internal validity as validation technic (Sargent 2013)
Several replications (runs) of a stochastic model are made to determine the amount of (internal) stochastic variability in the model
If there are large variation in the results it might make the model and the results questionable
Operational graphics as validation technic (Sargent 2013)
Performance measure are shown graphically while the model is running through time
The performance measure could be number un queue or percentage og servers which are busy
Parameter variability-sensitivity analysis as validation technic (Sargent 2013)
Changing values and input or internal parameter of the model to determine the effect at the model´s behavior or output.
The same relationship should occur in the model as in the real system.
Predictive validation as validation technic (Sargent 2013)
The model predicts (forecast) the system´s behavior. A comparison is made between these to see if they are the same.
Conceptual model validation according to Sargent (2013)
1) The theories and assumptions for the conceptual model are correct → Mathematical analysis and statistical methods on problem entity data.
2) The model’s representation of the problem entity and the model’s structure is reasonable for the intended purpose of the model → Has the appropriate detail and aggregate level been used?
Primary validation techniques are
- Face validation, structured walkthrough and trace
Computerized model verification according to Sargent (2013)
Ensures that the computer programming and implementation of the conceptual model are correct
Improving the verification can be done by using a simulation language
Primary technics are Structured walkthroughs and trace
Approached for testing simulation software (Sargent 2013)(Fairly 1976)
Static testing: the computer program is analysed to see if it is correct
- Structured walkthrough, correctness proofs and examining the structure properties of the program
Dynamic testing: The computer program is executed under different conditions and the values obtained are used to decide if the program and its implementation are correct
- Trace, input-output relationships, data relationship correctness
Operational validity according to Sargent (2013)
Operational validation is determining if the simulation model’s output behavior has the accuracy required for the intended purpose of the model
Major factor for this validity is whether the problem entity (the system) is observable
- Observable means possible to collect data on the operational behavior of the entity
Explore model behavior (operational validation Sargent 2013))
Can be done either qualitatively or quantitatively
In qualitatively analysis the direction of the output behavior are examined In quantitatively analysis both the direction and the precise magnitude of the output behavior is examined
Model exploration can be done using many of the technics we have had about, but parameter variability -sensitivity analysis is one to be used.
Comparisons of output behaviors (operational validation Sargent 2013)
Three basic approaches in comparing the simulation model output behavior to either the system or another model (output behavior)
- Hypothesis test (Objective decision)
- Confidence intervals (Objective)
- Graphs (Subjective)
Hypothesis test (Operational validation Sargent 2013)
Objective
Can be used in comparison of means, variance, distributions and time series of the output variables of a model to determine if the simulation model’s output behavior has a satisfactory range of accuracy
First hypothesis should be stated in order for them to be tested
Two possible errors when using hypothesis test
- Type I: rejecting the validity of a valid model – model builder’s risk
- Type II: accepting the validity of a invalid model – model user’s risk
Confidence intervals (Operational validation Sargent 2013)
Objective
Can be obtained for the differences between means, variances and distributions for variables
The confidence intervals can be used as the model range of accuracy for model validation
Graphical comparisons of data (Operational validation Sargent 2013)
Subjective decision is made on each graph whether it is within the accuracy is within the acceptable range
Data of the simulation model and system output variables are graphed for different experimental conditions to decide if the models output behavior has sufficient accuracy
Three types of graph are used
- Histograms
- Boxplot
- Behavior graphs (Scatter plot to show relationship)
Here it is also possible to make Type I and type II errors like in hypothesis test.
Recommended procedure for validation (Sargent 2013)
1) Make agreement between model development team, model sponsors and users about the decision making approach and specific validation technics to be used
2) Specify the acceptable range of accuracy – should be done before development of model or early in the development
3) Test assumptions and theories underlying the model
4) In every iteration perform face validity on conceptual model
5) Every model iteration explore the simulation model behavior using computerized model
6) At least in last model iteration, make comparisons if possible between model and system
7) Prepare verification documentation
8) If the model is to be used over time develop a schedule for periodic review of the model validity
Objective vs subjective use of validation technics (Sargent 2013)
Objective: using mathematical procedure or statistical test
- For instant hypothesis test or confidence intervals
Subjective: When a subjective decision is made whether the validity is acceptable
- For instant using graphs which rely on subjective conclusion and evaluation
Validation characterized (managerial perspective)
The problem validation from a managerial perspective - characterized by the following properties:
1. The processes of modeling and validation cannot be done separately
2. Consider model validation as a set of interconnected decisions to be made
3. To ensure the usefulness dimension of validity, an active participation of the main actors in the modeling-validating process and the consideration of the problem context in which this is carries out are emphasized.
Model assessors
A group independent of both model builders and model users → Necessitates a more explicit integration of model validation and documentation into the process of OR studies.
What does model validation involve?
Involves usefulness, usability, and cost considerations and representativeness within the context of a modeler-user interface
Purpose of the modeling-validation process
Aims to link the processes of the model building and the model validation into a single process
Why is it important to identify the stakeholder (users of the model) in the model-validating process?
It is a decision-system that involves human factors so the model involves value. In order to determine the favor values for the model the stakeholders should be determined early in the process.
The success or failure of a model depends very much on the attitude and behavior of stakeholders (the model’s clients)
The types of model objective
1)Prediction: Predict future values (or behavior) of some variables without necessarily desiring to understand the internal functioning of the system.
2)Comprehension: Better understand the problem situation without necessarily desiring to predict its future behavior.
Explain the importance-certainty graph
Consider the degree of importance and certainty of each relevant assumption.
Useful in providing a deeper insight into the problem situation, → forces model builders and decision-makers to clarify and compare the relevant assumptions with respect to each other → The model builder is then in a better position to judge where to put effort
What is meant by model confidence?
The user’s total attitude toward the model, and of the willingness to employ its results in making decisions.
Decision makers need to have confidence in the model and its output before using is for making decisions
Which costs are associated with a model:
Two types: 1) Cost of developing the model and 2) Cost of using and implementing the model
These two costs are usually in conflict their sum is a convex function of the level of validity. The appropriate level of validity is the one which minimizes the sum of the associated costs
Turing test:
Individuals who are knowledgeable about the
operations of the system being modelled are asked whether they can discriminate between system and model outputs.
Block box validation
In black-box validation, the model’s internal construction is treated as being unknown
The validation focuses on the predictive power of the model
The validation is based on a comparison of two sets of observations:
- one set from the model,
- the other from a reference system (the real system).
Statistical tests can be applied in order to assess the similarity between the two sets of observations
White/open box validation
In open-box validation, the detailed internal structure of the model is compared with that of its reference system (the real system)
