CIA - Models Flashcards
Define ‘model’.
It is a practical representation of relationships among entities, using FEMS concepts:
- Financial
- Economical
- Mathematical
- Statistical
What are the elements of a model? (3) (hint: SIR)
- model Specification
- model Implementation
- model Run
Define ‘model specification’.
It is a description of the parts of a model and their interactions, including:
- data
- assumptions
- methods
- entities
- events
Define ‘model implementation’.
It is the systems that perform the calculations:
- computer programs
- spreadsheets
- etc.
Define ‘model run’.
It is the inputs and outputs of the implementation.
Define ‘model risk’.
It is the risk that the user will draw inappropriate conclusions due to the shortcomings of the model or its use
What is the main distinction between a calculation and model?
That a model requires more documentation:
- how it was chosen
- how it is used
Why is there always risk in using a model?
It is because a model is a simplification of reality.
How can model risk be measured? (2)
- severity of model failure
- likelihood of model failure
Describe the considerations in assessing the severity of a model failure. (3)
- financial significance (eg: the severity is higher if estimating a major balance sheet item)
- importance of model (eg: severity is lower if multiple models are being used)
- frequency of use of model (eg: severity is higher if model is used frequently)
Describe the considerations in assessing the likelihood of model failure. (4)
- complexity (eg: higher complexity means a higher likelihood of misuse of the model)
- expertise (eg: non-expert users may not understand the model’s limitations)
- documentation (eg: bad documentation means that there is a higher likelihood that there will be a model failure)
- testing (eg: inadequate testing means that there is a higher likelihood of model failure)
Does the actuary have more control over the SEVERITY or the LIKELIHOOD of model failure? Justify your answer.
The actuary has more control over the LIKELIHOOD:
it is within the actuary’s control to:
- CHOOSE a more reliable model
- TEST the model more thoroughly
Identify the steps an actuary should take before using a new model. (4)
- review specifications
- validate implementation
- deal with the limitations
- keep documentation
Describe what an actuary does when reviewing a model’s specifications. (3) (hint: DAMs)
Verify that the :
- DATA fits the model requirements
- METHODS are sound
- ASSUMPTIONS are appropriate
Describe what an actuary does when validating a model’s implementations. (5)
- compare with other tested models
- maintain a set of test cases
- backtesting (testing with historical data where you already know the answer)
- run an entire live file through successive version of the model (for models with a higher risk-rating)
- peer review of testing procedure
Describe what an actuary does when dealing with a model’s limitations.
The actuary must understand the range of uses for which the model was designed and tested.
Describe what an actuary should include when documenting a model. (3)
- how the model was CHOSEN
- how it was TESTED
- what are its LIMITATIONS
What is an important tool for validating models?
A model’s risk rating (riskier models need more thorough validation).
How should an actuary evaluate an existing model that’s being used in a NEW way? (2)
- check that the initial model was properly validated
- review limitations in the new application that may not have been relevant in the initial application
How should an actuary evaluate a model approved by the use BY OTHERS?
The actuary should review & approve the initial validation report.
How should an actuary evaluate a model OUTSIDE the ACTUARY’S EXPERTISE? (5)
Make a reasonable attempt at understanding the model’s:
- specifications
- validation (extent to which experts were involved)
- risk-rating
- complexity
- control framework
Give an example of a model outside of the actuary’s expertise.
A credit-scoring model.
What is the purpose of sensitivity testing regarding models? (3)
- to validate a model
- to understand the relationship between inputs and outputs
- to develop a sense of comfort
How can model assumptions be tested in the context of sensitivity-testing? (3)
- test assumptions OUTSIDE the expected range
- test assumptions singly and then IN COMBINATION
- test assumptions with a NON-LINEAR relationship between inputs and outputs
What types of validation should be done when USING a model? (3)
Validation of Data, Assumptions and Results:
- data should be Reliable and Sufficient
- validate non-global assumptions that vary by model run
- results should be ‘reasonable’ relative to the input
What does it mean for data to be RELIABLE? (2)
- data RECONCILES to audited sources (eg: Balance Sheet
- data is REASONABLE with respect to prior period data (eg: data from prior quarter shouldn’t be hugely different)
What does it mean for data to be SUFFICIENT? (2)
- data FITS model specification
- data is available in a CONSISTENT format
What checks can be done to validate the results of a model? (2)
- inputs and outputs should be CONSISTENT (input data should match similar fields in output file)
- results should be REASONABLE in both magnitude & direction (ie: small change in inputs causes a small change in outputs)
Compare a uni-dimensional model risk rating with a two-dimensional model risk rating.
uni-dimensional approach:
- rating from 1-20 (20 is high)
- based on financial significance, complexity, expertise of users, and documentation
two-dimensional approach:
- assessed separately for severity & likelihood of failure
- final rating is a balance of these
