D - CIA Models

Question 1

CIA MODELS

define MODEL

identify 3 elements of a model

Answer 1

PRACTICAL REPRESENTATION OF RELATIONSHIPS AMONG ENTITIES using FEMS (Financial, Economical, Mathematical, Statistical) concepts

SIR
model SPECIFICATION
model IMPEMENTATION
model RUN

Question 2

CIA MODELS

Describe 3 elements of a model

Answer 2

SIR

SPECIFICATION
description of the parts of a model and their interactions (incl. data, assumptions, methods, entities, events)

IMPLEMENTATION
the systems that perform the calculations (computer programs, spreadsheets,…)

RUN
the inputs and outputs of the implementation

Question 3

CIA MODELS

define MODEL RISK

Answer 3

risk that the user will draw inappropriate conclusions due to shortcoming of the model or its use

Question 4

CIA MODELS

Main distinction between a calculation and a model

Answer 4

MODEL requires more documentation (how it was chosen, how it is used)

Question 5

CIA MODELS

How can model risk be measured

Answer 5

SEVERITY of model failure

LIKELIHOOD of model failure

Question 6

CIA MODELS

3 considerations in assessing SEVERITY OF MODEL FAILURE

4 consideration in assessing LIKELIHOOD OF MODEL FAILURE

Answer 6

—severity— (FIF)
FINANCIAL SIGNIFICANCE
(severity is higher if estimating a major balance sheet item)
IMPORTANCE OF MODEL
(severity is lower if multiple models are being used)
FREQUENCY OF USE OF MODE
(severity is higher if model is used frequently)

—likelihood—
COMPLEXITY
(higher complexity means higher likelihood of misuse of model)
EXPERTISE
(non-expert users may not understand model limitations)
DOCS
(bad docs means high likelihood of model failure)
TESTING
(inadequate testing means high likelihood of model failure)

Question 7

CIA MODELS

Why actuary have more control over the LIKELIHOOD than the SEVERITY of model failure

Answer 7

because actuary can CHOOSE a more reliable model (within the actuary’s control)

because actuary can TEST the model more thoroughly

Question 8

CIA MODELS

4 steps an actuary should take before using a new model

Answer 8

REVIEW SPECIFICATION
verify DAMs (data fit model requirement, methods are sound, assumptions are appropriate)

VALIDATE IMPLEMENTATION

• compare with other tested models
• maintain a set of test cases
• backtesting (testing with historical data where you already know the answer)
• run an entire live file through successive versions of the model (for model with a higher risk-rating)
• peer review of testing procedure

DEAL WITH LIMITATIONS
-understand the range of uses for which the model was designed and tested

KEEP DOCUMENTATION

• how the model was chosen
• how it was tested
• what are its limitations

Question 9

CIA MODELS

provide an important tool for validating models

Answer 9

a model’s RISK RATING

riskier models need more thorough validation

Question 10

CIA MODELS

how should an actuary evaluate :

• an existing model that’s being used in a NEW WAY
• a model approved for use BY OTHERS
• a model OUTSIDE ACTUARY’S EXPERTISE

Answer 10

NEW WAY
-check that the initial model was properly validated
-review limitations in the new application that may not have been relevant in the initial application
BY OTHERS
-actuary should review and approve the initial validation report
OUTSIDE ACTUARY’S EXPERTISE
-make a reasonable attempt at understanding the model’s specifications, validation, risk-rating, complexity, control framework

Question 11

CIA MODELS

example of model outside actuary’s expertise

Answer 11

a credit-scoring model

Question 12

CIA MODELS

purpose of sensitivity testing regarding models

Answer 12

• to validate a model
• to understand the relationship between inputs and outputs
• to develop a sense of comfort with the model

Question 13

CIA MODELS

How can model assumptions be tested in the context of sensitivity-testing?

Answer 13

• test assumptions OUTSIDE the expected range
• test assumptions separately and then IN COMBINATION
• test assumptions with a NONLINEAR relationship between inputs and outputs

Question 14

CIA MODELS

3 types of validations that should be done when using a model

Answer 14

DAR : validation of Data, Aussmptions, Results

Data should be “RelStuff” (reliable and sufficient)

validate non-global assumptions that vary by model run

results should be REASONABLE relative to input

Question 15

CIA MODELS

What does it mean for data to be:
RELIABLE?
SUFFICIENT?

Answer 15

–reliable–
data RECONCILES to audited sources (eg. balance sheet)

data is REASONABLE when compared to prior period data

–sufficient–
data FITS model specification

data is available in a CONSISTENT format

Question 16

CIA MODELS

what checks can be done to validate the results of a model

Answer 16

• inputs and outputs should be CONSISTENT (input data should match similar fields in output file)
• results should be REASONABLE in both magnitude and direction (small change in inputs causes a small change in outputs)

Question 17

CIA MODELS

Risk-Rating :
is severity OR likelihood of failure impacted (and state level of impact)

1) DOCS scribbled on a napkin
2) model used ONCE every 5 years
3) model VALUES 90% of business
4) user is stupid
5) model is streamlined for ease of use
6) model used to price premiums for 2% of book
7) model used to determine selling price of company

Answer 17

1) impacts likelihood :
high-risk
2) impacts likelihood : low-risk
3) impacts severity : high-risk
4) impacts likelihood : high-risk
5) impacts likelihood : low-risk
6) impacts severity : low-risk
7) impacts severity : high-risk

Question 18

CIA MODELS

compare uni-dimensional model risk-rating with two-dimensional rating

Answer 18

uni-dimensional approach:

• rating from 1-20 (20 is high)
• based on financial significance, complexity, expertise of users, docs

two-dimensional approach

• assessed separately for severity and likelihood of failure
• final rating is a balance of these