D - CIA Models Flashcards

1
Q

CIA MODELS

Define MODEL

A

Practical representation of relationships among entities using FEMS (Financial, Economical, Mathematical, Statistical) concepts

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

CIA MODELS

Describe 3 elements of a model

A
  • SPECIFICATION
    • Description of the parts of a model and their interactions (incl. data, assumptions, methods, entities, events)
  • IMPLEMENTATION
    • The systems that perform the calculations (computer programs, spreadsheets,…)
  • RUN
    • The inputs and outputs of the implementation
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

CIA MODELS

Define MODEL RISK

A

Risk that the user will draw inappropriate conclusions due to shortcoming of the model or its use

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

CIA MODELS

Main distinction between a calculation and a model

A

MODEL requires more documentation (how it was chosen, how it is used)

  • Models examples:
    • GLM
    • Creation of LDF to estimate Ult.
    • DCAT
  • Non-Model examples:
    • Least-Square regression (part of a model)
    • Spreadsheet used to summarize and reformat information
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

CIA MODELS

Why is there always risk in using a model?

A

Because a model is a simplification of reality.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

CIA MODELS

How can model risk be measured?

A
  1. SEVERITY of model failure
  2. LIKELIHOOD of model failure
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

CIA MODELS

3 considerations in assessing SEVERITY of model failure

A

FIF

  • Financial significance
    • Severity is higher if estimating a major balance sheet item
  • Importance of model
    • Severity is lower if multiple models are being used
  • Frequency of use of mode​l
    • Severity is higher if model is used frequently
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

CIA MODELS

4 consideration in assessing LIKELIHOOD of model failure

A
  • Complexity
    • Higher complexity means higher likelihood of misuse of model
  • Expertise
    • Non-expert users may not understand model limitations
  • Docs
    • Bad docs means high likelihood of model failure
  • Testing
    • Inadequate testing means high likelihood of model failure
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

CIA MODELS

Why actuary have more control over the LIKELIHOOD than the SEVERITY of model failure?

A

Because actuary can:

  1. CHOOSE a more reliable model (within the actuary’s control)
  2. TEST the model more thoroughly (within the actuary’s control)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

CIA MODELS

4 steps an actuary should take before using a new model

A
  1. Review specification
  2. Validate implementation
  3. Deal with limitations
  4. Keep documentation
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

CIA MODELS

Describe what an actuary does when reviewing a model’s SPECIFICATIONS (3)

A

Verify DAMs:

  1. Data fits the model requirements
  2. Assumptions are appropriate
  3. Methods are sound
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

CIA MODELS

Describe what an actuary does when validating a model’s IMPLEMENTATION (5)

A
  1. Compare with other tested models
  2. Maintain a set of test cases
  3. Backtesting (testing with historical data where you already know the answer)
  4. Run an entire live file through successive versions of the model (for model with a higher risk-rating)
  5. Peer review of testing procedure
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

CIA MODELS

Describe what an actuary does when dealing with a model’s LIMITATIONS

A

Understand the range of uses for which the model was designed and tested

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

CIA MODELS

Describe what an actuary should include when DOCUMENTING a model (3)

A
  1. How the model was chosen
  2. How it was tested
  3. What are its limitations
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

CIA MODELS

Provide an important tool for validating models

A

A model’s RISK RATING (riskier models need more thorough validation)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

CIA MODELS

How should an actuary evaluate :

  • An existing model that’s being used in a NEW WAY?
  • A model approved for use BY OTHERS?
  • A model OUTSIDE ACTUARY’S EXPERTISE?
A
  • NEW WAY
    • Check that the initial model was properly validated
    • Review limitations in the new application that may not have been relevant in the initial application
  • BY OTHERS
    • Actuary should review and approve the initial validation report
  • OUTSIDE ACTUARY’S EXPERTISE
    • Make a reasonable attempt at understanding the model’s:
      • Specifications
      • Validation
      • Risk-rating
      • Complexity
      • Control framework
17
Q

CIA MODELS

Example of model outside actuary’s expertise

A

a credit-scoring model

18
Q

CIA MODELS

Purpose of sensitivity testing regarding models (3)

A
  1. To validate a model
  2. To understand the relationship between inputs and outputs
  3. To develop a sense of comfort with the model
19
Q

CIA MODELS

How can model assumptions be tested in the context of sensitivity-testing?

A
  • Test assumptions OUTSIDE the expected range
  • Test assumptions separately and then IN COMBINATION
  • Test assumptions with a NONLINEAR relationship between inputs and outputs
20
Q

CIA MODELS

3 types of validations that should be done when using a model

A

DAR: validation of Data, Assumptions & Results

  • Data should be Reliable & Sufficient
  • Validate non-global assumptions that vary by model run
  • Results should be “reasonable” relative to input
21
Q

CIA MODELS

What does it mean for data to be:

  • Reliable?
  • Sufficient?
A
  • Reliable
    • Data RECONCILES to audited sources (eg. balance sheet)
    • Data is REASONABLE when compared to prior period data (YoY, QoQ)
  • Sufficient
    • Data FITS model specification
    • Data is available in a CONSISTENT format
22
Q

CIA MODELS

What checks can be done to validate the results of a model?

A
  • Inputs & outputs should be CONSISTENT
    • Input data should match similar fields in output file
  • Results should be REASONABLE in both magnitude and direction
    • Small change in inputs causes a small change in outputs
23
Q

CIA MODELS

Risk-Rating : Is severity OR likelihood of failure impacted (and state level of impact)?

  1. DOCS scribbled on a napkin
  2. Model used ONCE every 5 years
  3. Model VALUES 90% of business
  4. User is stupid
  5. Model is streamlined for ease of use
  6. Model used to price premiums for 2% of book
  7. Model used to determine selling price of company
A
  1. Impacts likelihood : high-risk
  2. Impacts severity : low-risk
  3. Impacts severity : high-risk
  4. Impacts likelihood : high-risk
  5. Impacts likelihood : low-risk
  6. Impacts severity : low-risk
  7. Impacts severity : high-risk
24
Q

CIA MODELS

Compare uni-dimensional model risk-rating with two-dimensional rating

A
  • Uni-dimensional approach:
    • Rating from 1-20 (20 is high)
    • Based on:
      • Financial significance
      • Importance of model
      • Complexity
      • Expertise of users
      • Docs
  • Two-dimensional approach
    • Assessed separately for severity and likelihood of failure
    • Final rating is a balance of these