Sargent (validation and verification) Flashcards
Why do validation:
In order to ensure that the model works for the intended purpose.
In problem structuring the purpose will often be to achieve clarity and to map interest and conflicting actions.
What is model verification according to Sargent (2013)?
Ensuring the computer programming and its implementation of the conceptual model are done correctly
What is model verification according to Sargent (2013)?
Ensuring the computer programming and its implementation of the conceptual model are done correctly
What is model validation according to Sargent (2013)?
Ensuring that the model serves it purpose
that a model works well and gives accurate results within its intended use.
What is model credibility according to Sargent (2013)?
Is concerned with developing confidence in the user, which is required to use the model and the derived information from the model
When is a model considered valid for experimental condition according to Sargent (2013)?
if the model’s accuracy falls within the desired range for its intended purpose, then it is considered acceptable.
Full validated model vs sufficient accuracy:
Often, we will not fully validate a model –> too costly.
Validate it to some extend. It will be sufficient accurate to the purpose of the model.
Doing more validation creating more confidence in the model also result in higher cost
The value of a model increases with the confidence in a model (validity) but at a decreasing rate
Three decision making approaches for deciding on simulation model validity (Sargent 2013)
All approached require the model development team to conduct verification and validation as part of the model development process
1) The model development team decide themselves if the model is valid
→ This is based on result of test and evaluations
2) The users of the simulation model is determining the validity of the model.
→ Better than above
→ Here the users are heavily part of the model development process when doing validation and verification
3) Use a third party to decide on the simulation models validity
→ Called an independent verification and validation (IV&V)
→ Often used when the model developed are very large
→ The IV&V team can conduct the validation during the development of model or after the model is developed.
- When done during the development team will not move on to the next stage before the IV&V team has evaluated it to satisfy the requirement for the current stage
- When done after the development of the model it is often extremely costly and time consuming
Explain the simplified version of the model development process (Sargent 1981):
Problem entity is the system or phenomena to be modelled
Conceptual model is the mathematical/logical/graphical representation of the problem
- Developed through analysis and modelling phase
Computerized model is the conceptual model implemented on a computer
- Developed through a computer programming and implementation phase
inferences about the problem entity are obtained by
conducting computer experiments on the computerized model in the experimentation phase.
Conceptual model validation is determining that theories and assumption underlying the conceptual model are correct and the model representation of the entity is reasonable for the intended purpose
Computerized model verification: Assuring that the computer programming and implementation of the conceptual model are correct
Operational validation determines that the models accuracy is within the range of accuracy for the models intended purpose
Data validity ensures that the data necessary for model building, model evaluation and test are adequate and correct
The process is iterative and every time any changes are made the process start over again
Animation as validation technic (Sargent 2013)
The model’s operational behavior is displayed graphically as the model moved trough time
For instant the movement of parts in a factory
Often not all behaviors being observed since it is often done in a short time interval
Comparison to other models as validation technic (Sargent 2013)
Various results of the simulation model is being validated by comparing it to results of other (valid) models
Data relationship correctness as validation technic (Sargent 2013)
Data relationship correctness validation ensures accurate and consistent connections between data elements in a model or system.
Degenerate tests as validation technic (Sargent 2013)
A test of the models behavior is done by appropriate selection of values of input and internal parameters.
For instant does the average number in the queue continue to increase over time, when the arrival rate is larger than the service rate ?
Event validity as validation technic (Sargent 2013)
The “events” of occurrences of the simulation model are compared to those of the real system to determine whether they are similar
For instant: compared the number of fires in the fire department simulation to the actual number of fires
Extreme condition test as validation technic (Sargent 2013)
The model structure and outputs should be plausible for any extreme and unlikely combination of levels of factors in the system
Face validity as validation technic (Sargent 2013)
Face validity is a subjective assessment of whether a model measures what it claims to measure based on its content and relevance.
For example is the logic in the conceptual model correct and are the models input-output relationship reasonable?
Historical data validation as validation technic (Sargent 2013)
If historical data exist, part of the data is used for build the model and remaining data are used to test whether the model behaves as the system does