CIA.Models Flashcards
define ‘Model’
a practical representation of relationships among entities or events using FEMS (Financial, Economical, Mathematical, Statistical) concepts
what are the elements of a model?
SIR
- Specification
- Implementation
- Runs
define ‘model specification’
a description of the parts of a model and their interactions (includes data, assumptions, methods, entities, events)
define ‘model implementation’
the systems that perform the calculations (computer programs, spreadsheets…)
define ‘model run’
the inputs and outputs of the implementation
define ‘model risk’
risk that user will draw inappropriate conclusions due to shortcomings of model use
what is the main distinction between a calculation and a model?
a model requires more documentation on how it was chosen and how it is used
why is there always risk in using a model?
because a model is a simplification of reality
3 strategies to mitigate model risk
- choose a model for a task
- oversee usage
- communicate model results
how can model risk be measured?
- severity of model failure
- likelihood of model failure
describe 4 considerations in assessing the severity of model failure
severity of rapping (need to ID rapper NF because he swears FU FS)
ID NF FU FS
- Impact of Decision being made using model
- Non-Financial impact (reputational)
- Frequency of Use
- Financial Significant of results model produces
describe 4 considerations in assessing the likelihood of model failure
how i met your mother (TED.C)
- Testing: sufficiency of testing
- Expertise: required level of expertise of users
- Documentation: adequacy of documentation
- Complexity: higher complexity means higher likelihood of misuse of model
does the actuary have more control over the severity or likelihood of model failure? Justify.
More control over likelihood through
- choose a more reliable model
- test the model more thoroughly
- tighter controls over model run
identify the 4 steps an actuary should take before using a new model
SLID
- review Specification
- deal with Limitations
- validate Implementation
- keep Documentation
describe what actuary does when reviewing a model’s specifications
verify DAM
- your Data fits the model requirements
- Assumptions are appropriate
- Methods are sound
describe what an actuary does when validating a model’s implementation
- compare with other tested models
- maintain a set of test cases
- back testing ( testing with historical data where you already know the answer)
- run an entire live file through successive versions of the model (for models with a higher risk rating)
- peer review of testing procedure
describe what an actuary does when dealing with model’s limitations
understand the range of uses of which the model was designed and tested
describe 3 items an actuary should include when documenting a model
- how the model was chosen
- how it was tested
- what are its limitations
what is an important tool for validating models?
a model’s risk rating
riskier models need more thorough validation
how should an actuary evaluate an existing model that’s being used in a new way?
- check that the initial model was properly validated
- review limitations in the new application that may not have been relevant in the initial application
how should an actuary evaluate a model approved for use by others
actuary should review and approve the initial validation report
how should an actuary evaluate a model outside actuary’s expertise
make a reasonable attempt at understanding the model’s
- specifications
- validation
- risk rating
- complexity
- control framework
give an example of a model outside actuary’s expertise
a credit scoring model
what is the purpose of sensitivity testing regarding models
validate comfort relationship
- validate model
- develop a sense of comfort within the model
- understand the relationship between inputs and outputs
compare sensitivity testing and scenario testing
sensitivity:
- incremental changes to risk factors
- shock is more immediate and shorter time horizon
- simpler
scenario:
- significant changes to risk factors
- observe future state including ripple effects and actions over longer time horizon
- more complex
how can model assumptions be tested in the context of sensitivity testing?
- outside expected range
- singly & in combination
- with a non linear relationship between inputs and outputs
what types of validation should be done when using a model
DAR
- Data should be RelSuff (reliable and sufficient)
- validate non global Assumptions that vary by model run
- Results should be reasonable relative to input
what does it mean for data to be reliable?
- data reconciles to audited sources (ex: balance sheet)
- data is reasonable with respect to prior period data
what does it mean for data to be sufficient?
- data fits model specification
- data is available in a consistent format
what checks can be done to validate the results of a model?
- inputs and outputs should be consistent
- results should be reasonable in both magnitude and direction
compare uni dimensional model risk rating with two dimensional rating
uni:
- rating from 1- 20 (20 is high)
- based on financial significant, complexity, expertise of users and documentation
two:
- assessed separately for severity and likelihood of failure
- final rating is a balance of these
