CIA.Models Flashcards
Define ‘model’
A practical representation of relationships among entities using FEMS concepts.
[Financial, Economical, Mathematical, Statistical]
What are the 3 elements of a ‘model’ (Hint = SIR)
- Specification
- Implementation
- Run
Define ‘model specification’
A description of the parts of a model and their interactions
(data, assumptions, methods, entities, events)
Define ‘model implementation’
The systems that perform the calculations
(computer programs, spreadsheets,…)
Define ‘model run’
The inputs/outputs of the implementation
Define ‘model risk’
The risk that the user will draw inappropriate conclusions due to shortcomings of the model or its use
What is the main distinction between a calculation and a model
A model requires more documentation (how it was chosen, how it’s used)
Why is there always risk in using a model?
Because a model is a simplification of reality
How can model risk be measured? (2)
- Severity of model failure
- Likelihood of model failure
Identify 3 considerations in assessing the severity of model failure
- Financial significance
(Ex: severity is higher if estimating a major balance sheet item) - Importance of model
(Ex: severity is lower if multiple models are being used) - Frequency of use of model
(Ex: severity is higher if model is used frequently)
Identify 4 considerations in assessing the likelihood of model failure
- Complexity
(Ex: higher complexity means higher likelihood of misuse of model) - Expertise
(Ex: non-expert users may not understand model limitations) - Docs
(Ex: bad docs means high likelihood of model failure!!!) - Testing
(Ex: inadequate testing means high likelihood of model failure)
Does the actuary have more control over the SEVERITY or LIKELIHOOD of model failure? (justify)
More control over likelihood
- CHOOSE a more reliable model (within the actuary’s control)
- TEST the model more thoroughly (within the actuary’s control)
Identify the 4 steps an actuary should take before using a new model
- Review specification
- Validate implementation
- Deal with limitations
- Keep documentation
Describe 3 things an actuary does when reviewing a model’s SPECIFICATIONS
Verify DAMs:
- your DATA fits the models requirements
- METHODS are sound
- ASSUMPTIONS are appropriate
Describe 5 things an actuary does when validating a model’s IMPLEMENTATION
- Compare with other tested models
- Maintain a set of test cases
- Backtesting (testing with historical data where you already know the answer)
- Run an entire live file through successive versions of the model (for models with a higher risk-rating)
- Peer review of testing procedure
Describe what an actuary does when dealing with model’s LIMITATIONS
Understand the range of uses for which the model was designed and tested
Identify 3 things an actuary should include when DOCUMENTING a model
- How the model was CHOSEN
- How it was TESTED
- What are its limitations
Identify an important tool for validating models
A model’s risk rating (riskier models need more thorough validation)
Identify 2 things an actuary should evaluate when looking at an existing model that’s being used in a NEW WAY
- Check that the initial model was properly validated
- Review limitations in the new application that may not have been relevant in the initial application
How should an actuary evaluate a model approved for use BY OTHERS?
Should review & approve the initial validation report
How should an actuary evaluate a model OUTSIDE ACTUARY’S EXPERTISE
Make a reasonable attempt at understanding the model’s…
- specifications
- validation (extent to which experts were involved)
- risk-rating
- complexity
- control framework
Give an example of a model outside actuary’s expertise
A credit-scoring model
What is the purpose (3) of sensitivity testing regarding models
- Validate model
- Understand the relationship between inputs/outputs
- Develop a sense of comfort with the model
How can model assumptions be tested in the context of sensitivity-testing? (3 ways)
- Test assumptions OUTSIDE the expected range
- Test assumptions singly and then COMBINED
- Test assumptions with a NONLINEAR relationship between inputs/outputs
Identify 3 types of validation that should be done when USING a model
(Hint: DAR)
DAR: validation of Data, Assumptions, Results
- data should be Reliable & Sufficient
- validate non-global assumptions that vary by model run
- results should be “reasonable” relative to input
What does it mean for data to be RELIABLE
- Data RECONCILES to audited sources (Ex: balance sheet)
- Data is REASONABLE with respect to prior period data (Ex: data from prior quarter shouldn’t be hugely different)
What does it mean for data to be SUFFICIENT
- Data FITS model specification
- Data is available in a CONSISTENT format
Identify 2 checks thant can be done to validate the results of a model
- Inputs/outputs should be CONSISTENT
(input data should match similar fields in output file) - Results should be REASONABLE in both magnitude & direction
(small change in inputs causes a small change in outputs)
Compare uni-dimensional model risk-rating with two-dimensional rating
Uni-dimensional approach:
- rating from 1-20 (20 is high)
- based on financial significance, complexity, expertise of users, docs
Two-dimensional approach:
- assessed separately for severity & likelihood of failure
- final rating is a balance of these
