D - CIA Models

Question 1

CIA MODELS

Define MODEL

Answer 1

Practical representation of relationships among entities using FEMS (Financial, Economical, Mathematical, Statistical) concepts

Question 2

CIA MODELS

Describe 3 elements of a model

Answer 2

• SPECIFICATION
  
  • Description of the parts of a model and their interactions (incl. data, assumptions, methods, entities, events)
• IMPLEMENTATION
  
  • The systems that perform the calculations (computer programs, spreadsheets,…)
• RUN
  
  • The inputs and outputs of the implementation

Question 3

CIA MODELS

Define MODEL RISK

Answer 3

Risk that the user will draw inappropriate conclusions due to shortcoming of the model or its use

Question 4

CIA MODELS

Main distinction between a calculation and a model

Answer 4

MODEL requires more documentation (how it was chosen, how it is used)

• Models examples:
  
  • GLM
  • Creation of LDF to estimate Ult.
  • DCAT
• Non-Model examples:
  
  • Least-Square regression (part of a model)
  • Spreadsheet used to summarize and reformat information

Question 5

CIA MODELS

Why is there always risk in using a model?

Answer 5

Because a model is a simplification of reality.

Question 6

CIA MODELS

How can model risk be measured?

Answer 6

1. SEVERITY of model failure
2. LIKELIHOOD of model failure

Question 7

CIA MODELS

3 considerations in assessing SEVERITY of model failure

Answer 7

FIF

• Financial significance
  
  • Severity is higher if estimating a major balance sheet item
• Importance of model
  
  • Severity is lower if multiple models are being used
• Frequency of use of mode​l
  
  • Severity is higher if model is used frequently

Question 8

CIA MODELS

4 consideration in assessing LIKELIHOOD of model failure

Answer 8

• Complexity
  
  • Higher complexity means higher likelihood of misuse of model
• Expertise
  
  • Non-expert users may not understand model limitations
• Docs
  
  • Bad docs means high likelihood of model failure
• Testing
  
  • Inadequate testing means high likelihood of model failure

Question 9

CIA MODELS

Why actuary have more control over the LIKELIHOOD than the SEVERITY of model failure?

Answer 9

Because actuary can:

1. CHOOSE a more reliable model (within the actuary’s control)
2. TEST the model more thoroughly (within the actuary’s control)

Question 10

CIA MODELS

4 steps an actuary should take before using a new model

Answer 10

1. Review specification
2. Validate implementation
3. Deal with limitations
4. Keep documentation

Question 11

CIA MODELS

Describe what an actuary does when reviewing a model’s SPECIFICATIONS (3)

Answer 11

Verify DAMs:

1. Data fits the model requirements
2. Assumptions are appropriate
3. Methods are sound

Question 12

CIA MODELS

Describe what an actuary does when validating a model’s IMPLEMENTATION (5)

Answer 12

1. Compare with other tested models
2. Maintain a set of test cases
3. Backtesting (testing with historical data where you already know the answer)
4. Run an entire live file through successive versions of the model (for model with a higher risk-rating)
5. Peer review of testing procedure

Question 13

CIA MODELS

Describe what an actuary does when dealing with a model’s LIMITATIONS

Answer 13

Understand the range of uses for which the model was designed and tested

Question 14

CIA MODELS

Describe what an actuary should include when DOCUMENTING a model (3)

Answer 14

1. How the model was chosen
2. How it was tested
3. What are its limitations

Question 15

CIA MODELS

Provide an important tool for validating models

Answer 15

A model’s RISK RATING (riskier models need more thorough validation)

Question 16

CIA MODELS

How should an actuary evaluate :

• An existing model that’s being used in a NEW WAY?
• A model approved for use BY OTHERS?
• A model OUTSIDE ACTUARY’S EXPERTISE?

Answer 16

• NEW WAY
  
  • Check that the initial model was properly validated
  • Review limitations in the new application that may not have been relevant in the initial application
• BY OTHERS
  
  • Actuary should review and approve the initial validation report
• OUTSIDE ACTUARY’S EXPERTISE
  
  • Make a reasonable attempt at understanding the model’s:
    
    • Specifications
    • Validation
    • Risk-rating
    • Complexity
    • Control framework

Question 17

CIA MODELS

Example of model outside actuary’s expertise

Answer 17

a credit-scoring model

Question 18

CIA MODELS

Purpose of sensitivity testing regarding models (3)

Answer 18

1. To validate a model
2. To understand the relationship between inputs and outputs
3. To develop a sense of comfort with the model

Question 19

CIA MODELS

How can model assumptions be tested in the context of sensitivity-testing?

Answer 19

• Test assumptions OUTSIDE the expected range
• Test assumptions separately and then IN COMBINATION
• Test assumptions with a NONLINEAR relationship between inputs and outputs

Question 20

CIA MODELS

3 types of validations that should be done when using a model

Answer 20

DAR: validation of Data, Assumptions & Results

• Data should be Reliable & Sufficient
• Validate non-global assumptions that vary by model run
• Results should be “reasonable” relative to input

Question 21

CIA MODELS

What does it mean for data to be:

• Reliable?
• Sufficient?

Answer 21

• Reliable
  
  • Data RECONCILES to audited sources (eg. balance sheet)
  • Data is REASONABLE when compared to prior period data (YoY, QoQ)
• Sufficient
  
  • Data FITS model specification
  • Data is available in a CONSISTENT format

Question 22

CIA MODELS

What checks can be done to validate the results of a model?

Answer 22

• Inputs & outputs should be CONSISTENT
  
  • Input data should match similar fields in output file
• Results should be REASONABLE in both magnitude and direction
  
  • Small change in inputs causes a small change in outputs

Question 23

CIA MODELS

Risk-Rating : Is severity OR likelihood of failure impacted (and state level of impact)?

1. DOCS scribbled on a napkin
2. Model used ONCE every 5 years
3. Model VALUES 90% of business
4. User is stupid
5. Model is streamlined for ease of use
6. Model used to price premiums for 2% of book
7. Model used to determine selling price of company

Answer 23

1. Impacts likelihood : high-risk
2. Impacts severity : low-risk
3. Impacts severity : high-risk
4. Impacts likelihood : high-risk
5. Impacts likelihood : low-risk
6. Impacts severity : low-risk
7. Impacts severity : high-risk

Question 24

CIA MODELS

Compare uni-dimensional model risk-rating with two-dimensional rating

Answer 24

• Uni-dimensional approach:
  
  • Rating from 1-20 (20 is high)
  • Based on:
    
    • Financial significance
    • Importance of model
    • Complexity
    • Expertise of users
    • Docs
• Two-dimensional approach
  
  • Assessed separately for severity and likelihood of failure
  • Final rating is a balance of these