CIA.Models

Question 1

Define “model”

Answer 1

A practical representation of relationships among entities using Financial, Economical, Mathematical, Statistical concepts

Question 2

What are the elements of a “model” (3)

Answer 2

SIR:
- Model Specification
- Model Implementation
- Model Run

Question 3

Define “model specification”

Answer 3

Is the description of the components of a model and the interrelationships of those components with each other, including the types of data, assumptions, methods, entities, and events

Question 4

Define “model implementation”

Answer 4

Is one or more systems developed to perform the calculation for a model specification

Question 5

Define “model run”

Answer 5

Is a set of inputs and the corresponding results produced by a model implementation

Question 6

Define “model risk”

Answer 6

Is the risk that, due to flaws or limitations in the model or in its use, the actuary or a user of the results of the model will draw an inappropriate conclusion from those results

Question 7

What is the main distinction between a calculation and a model

Answer 7

A model requires more documentation (how it was chosen, how it’s used)

Question 8

Why is there always risk in using a model

Answer 8

Because a model is a simplification of reality

Question 9

Name 3 strategies employed by actuaries when mitigating model risk

Answer 9

• Choose a model for a task
• Use the model (one-time or ongoing) or oversee its usage
• Communicate results of that model

Question 10

How can model risk be measured (2)

Answer 10

• Severity of model failure (how bad can it be)
• Frequency of model failure

Question 11

Describe considerations in assessing the severity of model failure (3)

Answer 11

• Financial significance (Ex: severity is higher if estimating a major balance sheet item)
• Importance of model (Ex: severity is lower if multiple models are being used)
• Frequency of use of model (Ex: severity is higher if model is used frequently)

Question 12

Describe considerations in assessing the likelihood of model failure (4)

Answer 12

• Complexity of model (ex: higher complexity means higher likelihood of misuse of model)
• Expertise of users (ex: non-expert users may not understand model limitations)
• Documentation of model (ex: bad documentation means high likelihood of model failure)
• Adequacy of testing (ex: inadequate testing means high likelihood of model failure)

Question 13

Does the actuary have more control over the SEVERITY or LIKELIHOOD of model failure (justify)?

Answer 13

More control over likelihood through:
- Choosing a better model
- Exercising greater care in validation
- Employing tighter controls for model runs

Question 14

Identify the steps an actuary should take before using a new model (4)

Answer 14

• Review specifications
• Validate implementation
• Deal with limitations
• Keep documentation

Question 15

Describe what an actuary does when reviewing a model’s SPECIFICATIONS

Answer 15

Verify DAMs:
- your DATA fits the models requirements
- ASSUMPTIONS are appropriate
- METHODS are sound

If using third party model, actuary will want to perform the appropriate tests to assess any important aspects not covered in the user’s documentation

Question 16

Describe what an actuary does when validating a model’s IMPLEMENTATION

Answer 16

• Compare with other tested models
• Maintain a set of test cases
• Backtesting (testing with historical data where you already know the answer)
• Run an entire live file through successive versions of the model (for models with a higher risk-rating)
• Peer review of testing procedure

Question 17

Describe what an actuary does when dealing with a model’s LIMITATIONS

Answer 17

• Understand the range of uses for which the model was designed and tested
• Actuary would be aware of which events are independent of each other and which are correlated
• Actuaries would be alert to assumptions that are fixed or embedded in a model

Question 18

Describe what an actuary should include when DOCUMENTING a model

Answer 18

• How the model was chosen
• How the model was tested
• What are the model’s limitations

Question 19

What is an important tool for validating models

Answer 19

A model’s risk rating (riskier models need more thorough validation)

Question 20

How should an actuary evaluate an existing model that’s being used in a NEW WAY

Answer 20

• Check that the initial model was properly validated
• Review limitations in the new application that may not have been relevant in the initial application

Question 21

How should an actuary evaluate a model approved for use BY OTHERS

Answer 21

Actuary should review & approve the initial validation report (if possible)

In some cases, the actuary may choose to rely on the validation done by others outside their firm.

Question 22

How should an actuary evaluate a model OUTSIDE THEIR EXPERTISE

Answer 22

Actuary would determine the appropriate level of reliance on experts, considering:
- If the individuals on whom the actuary is relying are considered experts in their field of practice
- The extend to which the model has been reviewed by experts in the applicable field
- The risk rating associated with the model

Actuary would make a reasonable attempt to understand:
- The basic workings of the model including its inputs, outputs, and general approach
- The testing and validation work that was completed
- The model’s complexity and the control framework used

Question 23

Give an example of a model outside actuary’s expertise

Answer 23

A credit-scoring model

Question 24

What is the purpose of sensitivity-testing regarding models

Answer 24

• To validate a model
• To understand the relationship between inputs/outputs
• To develop a sense of comfort with the model

Question 25

How can model assumptions be tested in the context of sensitivity-testing

Answer 25

• Test assumptions OUTSIDE expected range
• Test assumptions singly and then IN COMBINATION
• Test assumptions with a NONLINEAR relationship between inputs/outputs

Question 26

What types of validation should be done when USING a model (3)

Answer 26

DAR: validation of Data, Assumptions, Results
- Data should be reliable and sufficient
- Validate non-global assumptions that vary by model run
- Results should be “reasonable” relative to input

Question 27

What does the actuary consider regarding data reliability (6)

Answer 27

1. Reconciliation to other sources (preferably audited)
2. Summarize and compare input data to prior periods, if applicable
3. Check and investigate data points that are outliers for possible errors.
4. How are missing data handled? Is a data assumption made or is an error generated?
5. Data assumptions would be reviewed periodically to assess their appropriateness
6. Is the size of the data file consistent with prior periods?

Question 28

What does the actuary consider regarding data sufficiency (2)

Answer 28

1. Do the data meet the requirements of the model specification?
2. If the model will be used repeatedly, are the data in a consistent format every time?

Question 29

What are the 3 considerations for reviewing model assumptions?

Answer 29

1. Regular peer review (internal/external) of the assumptions
2. Are the intended assumptions the ones used in the model?
3. Are model assumptions unchanged unless they were meant to be changed?

Question 30

What checks can be done to validate the results of a model?

Answer 30

1. Are outputs consistent with inputs? (for ex: do the total amount of policies match)
2. How many errors were generated and what amount was involved? It is within established tolerance?
3. Are results as expected, both in direction and magnitude?

Question 31

What will the actuary review when validating the results of a stochastic model?

Answer 31

1. The results from a carefully chosen sample of realized deterministic scenarios, covering an appropriate range of inputs and/or assumptions
2. The distribution of output results for reasonability, paying particular attention to items such as trend, mean, median, etc.
3. Whether the results of the chosen deterministic scenarios are consistent with the distribution of stochastic results

Question 32

Is severity or likelihood of failure impacted ( & state level of impact): Documentation scribbled on napkin

Answer 32

Impacts likelihood of failure : HIGH-RISK

Question 33

Is severity or likelihood of failure impacted ( & state level of impact): model used ONCE every 5 years

Answer 33

Impacts severity of failure: LOW RISK

Question 34

Is severity or likelihood of failure impacted ( & state level of impact): model values 90% of business

Answer 34

Impacts severity of failure: HIGH RISK

Question 35

Is severity or likelihood of failure impacted ( & state level of impact): User is very inexperienced

Answer 35

Impacts likelihood of failure: HIGH RISK

Question 36

Is severity or likelihood of failure impacted ( & state level of impact): model is STREAMLINED for ease of use

Answer 36

Impacts likelihood of failure: LOW-RISK
- Could also argue it presents severity risk for being too simple (may omit important variables/interactions)

Question 37

Is severity or likelihood of failure impacted ( & state level of impact): model used to price PREMIUMS for 2% of book (by premium)

Answer 37

Impacts severity of failure: LOW RISK

Question 38

Is severity or likelihood of failure impacted ( & state level of impact): model used to determine SELLING PRICE of company

Answer 38

Impacts severity of failure: HIGH RISK

Question 39

Compare uni-dimensional model risk-rating with two-dimensional rating

Answer 39

Uni dimensional approach:
- Rating from 1-20 (20 is high)
- Based on financial significance, complexity, expertise of users, documentation

Two-dimensional approach:
- Assessed separately for severity & likelihood of failure
- Final rating is a balance of these