D - CIA Models Flashcards
CIA MODELS
Define MODEL
Practical representation of relationships among entities using FEMS (Financial, Economical, Mathematical, Statistical) concepts
CIA MODELS
Iidentify 3 elements of a model
SIR
- model SPECIFICATION
- model IMPEMENTATION
- model RUN
CIA MODELS
Describe 3 elements of a model
- SPECIFICATION
- Description of the parts of a model and their interactions (incl. data, assumptions, methods, entities, events)
- IMPLEMENTATION
- The systems that perform the calculations (computer programs, spreadsheets,…)
- RUN
- The inputs and outputs of the implementation
CIA MODELS
Define MODEL RISK
Risk that the user will draw inappropriate conclusions due to shortcoming of the model or its use
CIA MODELS
Main distinction between a calculation and a model
MODEL requires more documentation (how it was chosen, how it is used)
- Models examples:
- GLM
- Creation of LDF to estimate Ult.
- DCAT
- Non-Model examples:
- Least-Square regression (part of a model)
- Spreadsheet used to summarize and reformat information
CIA MODELS
Why is there always risk in using a model?
Because a model is a simplification of reality.
CIA MODELS
How can model risk be measured?
- SEVERITY of model failure
- LIKELIHOOD of model failure
CIA MODELS
3 considerations in assessing SEVERITY of model failure
FIF
- Financial significance
- Severity is higher if estimating a major balance sheet item
- Importance of model
- Severity is lower if multiple models are being used
- Frequency of use of model
- Severity is higher if model is used frequently
CIA MODELS
4 consideration in assessing LIKELIHOOD of model failure
- Complexity
- Higher complexity means higher likelihood of misuse of model
- Expertise
- Non-expert users may not understand model limitations
- Docs
- Bad docs means high likelihood of model failure
- Testing
- Inadequate testing means high likelihood of model failure
CIA MODELS
Why actuary have more control over the LIKELIHOOD than the SEVERITY of model failure?
Because actuary can:
- CHOOSE a more reliable model (within the actuary’s control)
- TEST the model more thoroughly (within the actuary’s control)
CIA MODELS
4 steps an actuary should take before using a new model
- Review specification
- Validate implementation
- Deal with limitations
- Keep documentation
CIA MODELS
Describe what an actuary does when reviewing a model’s SPECIFICATIONS (3)
Verify DAMs:
- Data fits the model requirements
- Assumptions are appropriate
- Methods are sound
CIA MODELS
Describe what an actuary does when validating a model’s IMPLEMENTATION (5)
- Compare with other tested models
- Maintain a set of test cases
- Backtesting (testing with historical data where you already know the answer)
- Run an entire live file through successive versions of the model (for model with a higher risk-rating)
- Peer review of testing procedure
CIA MODELS
Describe what an actuary does when dealing with a model’s LIMITATIONS
Understand the range of uses for which the model was designed and tested
CIA MODELS
Describe what an actuary should include when DOCUMENTING a model (3)
- How the model was chosen
- How it was tested
- What are its limitations
CIA MODELS
Provide an important tool for validating models
A model’s RISK RATING (riskier models need more thorough validation)
CIA MODELS
How should an actuary evaluate :
- An existing model that’s being used in a NEW WAY?
- A model approved for use BY OTHERS?
- A model OUTSIDE ACTUARY’S EXPERTISE?
- NEW WAY
- Check that the initial model was properly validated
- Review limitations in the new application that may not have been relevant in the initial application
- BY OTHERS
- Actuary should review and approve the initial validation report
- OUTSIDE ACTUARY’S EXPERTISE
- Make a reasonable attempt at understanding the model’s:
- Specifications
- Validation
- Risk-rating
- Complexity
- Control framework
- Make a reasonable attempt at understanding the model’s:
CIA MODELS
Example of model outside actuary’s expertise
a credit-scoring model
CIA MODELS
Purpose of sensitivity testing regarding models (3)
- To validate a model
- To understand the relationship between inputs and outputs
- To develop a sense of comfort with the model
CIA MODELS
How can model assumptions be tested in the context of sensitivity-testing?
- Test assumptions OUTSIDE the expected range
- Test assumptions separately and then IN COMBINATION
- Test assumptions with a NONLINEAR relationship between inputs and outputs
CIA MODELS
3 types of validations that should be done when using a model
DAR: validation of Data, Assumptions & Results
- Data should be Reliable & Sufficient
- Validate non-global assumptions that vary by model run
- Results should be “reasonable” relative to input
CIA MODELS
What does it mean for data to be:
- Reliable?
- Sufficient?
- Reliable
- Data RECONCILES to audited sources (eg. balance sheet)
- Data is REASONABLE when compared to prior period data (YoY, QoQ)
- Sufficient
- Data FITS model specification
- Data is available in a CONSISTENT format
CIA MODELS
What checks can be done to validate the results of a model?
- Inputs & outputs should be CONSISTENT
- Input data should match similar fields in output file
- Results should be REASONABLE in both magnitude and direction
- Small change in inputs causes a small change in outputs
CIA MODELS
Risk-Rating : Is severity OR likelihood of failure impacted (and state level of impact)?
- DOCS scribbled on a napkin
- Model used ONCE every 5 years
- Model VALUES 90% of business
- User is stupid
- Model is streamlined for ease of use
- Model used to price premiums for 2% of book
- Model used to determine selling price of company
- Impacts likelihood : high-risk
- Impacts severity : low-risk
- Impacts severity : high-risk
- Impacts likelihood : high-risk
- Impacts likelihood : low-risk
- Impacts severity : low-risk
- Impacts severity : high-risk
CIA MODELS
Compare uni-dimensional model risk-rating with two-dimensional rating
- Uni-dimensional approach:
- Rating from 1-20 (20 is high)
- Based on:
- Financial significance
- Importance of model
- Complexity
- Expertise of users
- Docs
- Two-dimensional approach
- Assessed separately for severity and likelihood of failure
- Final rating is a balance of these
