Verifiation, Validation, and Iteration

Question 1

Describe the concept of verification in computer modelling.

Answer 1

We need to be confident that the model is not wrong. Watch out for silent bugs. Use unit testing, benchmarking, and Davids Gold Secret

Question 2

What is a silent bug?

Answer 2

The code is running incorrectly and you don’t realise there is an error (e.g mathematical implementation).

Question 3

What is unit testing?

Answer 3

A function that checks another function within the code is implemented correctly. Hand calculate a value and assert an error if the difference is above the threshold.

Question 4

What is benchmarking?

Answer 4

Solve the ODE analytically under simplified conditions. Verify that the analytical and numerical solutions are similar for the same parameters. Error analysis and time step convergence.

Question 5

What is David’s Gold Secret?

Answer 5

Blow up one parameter so the physics dominates, or reduce a parameter so the physics disappears. What do you expect to happen? Does is happen?

Question 6

What is validation?

Answer 6

Does my model represent reality? Assume the model is technically correct.

Question 7

What do we have to accept to undertake validation?

Answer 7

That all model are flawed, imperfect representations of reality. We accept that it is not possible to prove a model is suitable under all possible scenarios. Use validation experiment or calibration.

Question 8

What is a validation experiment?

Answer 8

Devise and conduct an experiment that the model should be able to replicate. Predict outcome by running model with the experimental parameters and compare the results. The real world is an example of a v.e that we can not control.

Question 9

What is calibration?

Answer 9

Can we choose parameters that the model can accurately replicate the available data? If a model can describe the future reality, it ought to be able to describe the past data.

Question 10

Model Iteration: What are the errors that can be changed relevant to the how and assume parts?

Answer 10

Model not correctly scoped or conceptualised, need more/different physics, wrong BCs, ICs, parameters.

Question 11

Model Iteration: What are the errors that can be changed relevant to the given part?

Answer 11

Don’t have enough data to constrain the model.

Question 12

Model Iteration: What are the errors that can be changed relevant to the formulate part?

Answer 12

Model takes too long to run (try a different method), model can’t resolve correct time/space scales.

Question 13

What is model iteration?

Answer 13

Making changes to a model in response to identified shortcomings in verification and validation.

Question 14

Describe why both verification and validation can be considered forms of quality assurance

Answer 14

They both ensure that the model is correct.

Question 15

Describe how physical intuition can be used to verify model behaviour

Answer 15

If the trend of the solution is negative when it should be positive (e.g adding mass and increasing temperature actually is decreasing) we can conclude that it is wrong.

Question 16

Describe how the concept of diminishing returns applies to model iteration

Answer 16

We as developers of models need to consider the tradeoff between accuracy of the model, and the computational time required to obtain that level of accuracy. When the change in accuracy is so small, we may be nearing the point of diminishing returns.

Question 17

What are some characteristics of a good benchmark solution?

Answer 17

Numerical and analytical are similar with the same parameters.

Question 18

How can benchmarking be used to select the timestep when solving a problem numerically?

Answer 18

Use timestep convergence to see at what time step the numerical becomes close enough to the analytical solution. You must choose a threshold and justify it.