Backtesting & Simulation

Question 1

What are 4 methods used to evaluate investment strategies?

Answer 1

• backtesting: how strategy performed if implemented in past
• historical (scenario analysis) stress testing: how strategy performs under certain conditions
• simulation: how strategy would perform in hypothetical scenarios rather than limiting it to actual historical scenarios
• sensitivity analysis: quantify impact of changing assumptions (variables) in a model

Question 2

What is the objective of backtesting?

Answer 2

• use backtesting to determine how their real life investment strategies would have performed using historical data, allows for managers to gain insights & improve their skills

Question 3

What 3 steps does backtesting process involve?

Answer 3

• strategy design
• historical investment simulation
• analysis of backtesting

Question 4

What are 7 key activities in the strategy design step of the backtesting process?

Answer 4

• specify investment hypothesis & goals
• determine investment rules & processes
• decide key parameters (ratios)
• investment universe
• return definition
• rebalancing frequency & transaction costs
• start & end date

Question 5

Describe the second step (historical investment simulation) of the backtesting process.

Answer 5

• build portfolio based on rules established in step 1 & simulate how it would have performed during historical period

Question 6

What is rolling window backtesting?

Answer 6

• take orginal sample, then see how orginal sample performs in different time frame and it becomes out of sample data, take out of sample data and test again for a new time frame and it becomes the in sample data (process is repeated over & over)

Question 7

Describe step 3 of the backtesting process (analysis of backtesting output).

Answer 7

• analyze return data and consider key metrics

Question 8

What are 3 common problems in backtesting?

Answer 8

• survivorship bias: not accounting for stocks dropped from indexes will have impact on backtesting
• look-ahead bias: information was unknown or unknowable during backtesting timeline
• data snooping bias: when data is overanalyzed which gives rise to statistically irrelevant & occasionally nonexistent patterns

Question 9

When performing historical scenario analysis what are 2 significant structural breaks you must consider? What are structural breaks in economic conditions?

Answer 9

• expansions vs recessions
• periods of high volatility vs periods of low volatility
• structural breaks: unexpected change over time in economic conditions

Question 10

What is sensitivity analysis?

Answer 10

• sensitivity analysis: quantify impact of changing assumptions (variables) in a model

Question 11

What are 2 types of simulations that seek to avoid backtesting drawbacks?

Answer 11

• historical simulation
• Monte Carlo simulation

Question 12

What are the 8 steps to a properly designed simulation?

Answer 12

• determine target variable (usually portfolios return)
• specify key decision variables (returns on individual assets & weights within portfolio)
• specify number of trials
• define distributional properties (eg. normal, log normal, binomial, etc)
• draw N random numbers for each key decision variable using random number generator
• compute value of target variable for each set of simulate decision variables
• repeat straps 5&6 until N trials have been complete
• with N returns you can calculate mean return, volatility, sharpe ratio, etc)