CIA.Models Flashcards
Define “model”
A practical representation of relationships among entities using Financial, Economical, Mathematical, Statistical concepts
What are the elements of a “model” (3)
SIR:
- Model Specification
- Model Implementation
- Model Run
Define “model specification”
Is the description of the components of a model and the interrelationships of those components with each other, including the types of data, assumptions, methods, entities, and events
Define “model implementation”
Is one or more systems developed to perform the calculation for a model specification
Define “model run”
Is a set of inputs and the corresponding results produced by a model implementation
Define “model risk”
Is the risk that, due to flaws or limitations in the model or in its use, the actuary or a user of the results of the model will draw an inappropriate conclusion from those results
What is the main distinction between a calculation and a model
A model requires more documentation (how it was chosen, how it’s used)
Why is there always risk in using a model
Because a model is a simplification of reality
Name 3 strategies employed by actuaries when mitigating model risk
- Choose a model for a task
- Use the model (one-time or ongoing) or oversee its usage
- Communicate results of that model
How can model risk be measured (2)
- Severity of model failure (how bad can it be)
- Frequency of model failure
Describe considerations in assessing the severity of model failure (3)
- Financial significance (Ex: severity is higher if estimating a major balance sheet item)
- Importance of model (Ex: severity is lower if multiple models are being used)
- Frequency of use of model (Ex: severity is higher if model is used frequently)
Describe considerations in assessing the likelihood of model failure (4)
- Complexity of model (ex: higher complexity means higher likelihood of misuse of model)
- Expertise of users (ex: non-expert users may not understand model limitations)
- Documentation of model (ex: bad documentation means high likelihood of model failure)
- Adequacy of testing (ex: inadequate testing means high likelihood of model failure)
Does the actuary have more control over the SEVERITY or LIKELIHOOD of model failure (justify)?
More control over likelihood through:
- Choosing a better model
- Exercising greater care in validation
- Employing tighter controls for model runs
Identify the steps an actuary should take before using a new model (4)
- Review specifications
- Validate implementation
- Deal with limitations
- Keep documentation
Describe what an actuary does when reviewing a model’s SPECIFICATIONS
Verify DAMs:
- your DATA fits the models requirements
- ASSUMPTIONS are appropriate
- METHODS are sound
If using third party model, actuary will want to perform the appropriate tests to assess any important aspects not covered in the user’s documentation
Describe what an actuary does when validating a model’s IMPLEMENTATION
- Compare with other tested models
- Maintain a set of test cases
- Backtesting (testing with historical data where you already know the answer)
- Run an entire live file through successive versions of the model (for models with a higher risk-rating)
- Peer review of testing procedure
Describe what an actuary does when dealing with a model’s LIMITATIONS
- Understand the range of uses for which the model was designed and tested
- Actuary would be aware of which events are independent of each other and which are correlated
- Actuaries would be alert to assumptions that are fixed or embedded in a model
Describe what an actuary should include when DOCUMENTING a model
- How the model was chosen
- How the model was tested
- What are the model’s limitations
What is an important tool for validating models
A model’s risk rating (riskier models need more thorough validation)
How should an actuary evaluate an existing model that’s being used in a NEW WAY
- Check that the initial model was properly validated
- Review limitations in the new application that may not have been relevant in the initial application
How should an actuary evaluate a model approved for use BY OTHERS
Actuary should review & approve the initial validation report (if possible)
In some cases, the actuary may choose to rely on the validation done by others outside their firm.
How should an actuary evaluate a model OUTSIDE THEIR EXPERTISE
Actuary would determine the appropriate level of reliance on experts, considering:
- If the individuals on whom the actuary is relying are considered experts in their field of practice
- The extend to which the model has been reviewed by experts in the applicable field
- The risk rating associated with the model
Actuary would make a reasonable attempt to understand:
- The basic workings of the model including its inputs, outputs, and general approach
- The testing and validation work that was completed
- The model’s complexity and the control framework used
Give an example of a model outside actuary’s expertise
A credit-scoring model
What is the purpose of sensitivity-testing regarding models
- To validate a model
- To understand the relationship between inputs/outputs
- To develop a sense of comfort with the model
