9. A Protocol in the Era of ML

Question 1

1. Motivation

Answer 1

1. Inputs are guided by a reasonable hypothesis
2. Hypothesis before looking at data!

(Important bc large out-of-the-sample tests not possible in ML)

Question 2

2. Multiple Testing and statistical methods

Answer 2

1. Keep track of tried strategies
2. Beware of parallel universe problem

( 2-sigma is not necessarily meaningful, and a strategy working by chance is not necessarily significant )

Question 3

3. Sample choice and data

Answer 3

1. Test sample defined and justified ex-Ante
   –> Justify in advance and never change sample
2. Winsorization (truncation at certain threshhold) defined and justified ex-Ante
3. Pre-processing to ensure data quality
4. Pre-processing choices regarding Data transformation or Outlier exclusion need to be justified and documented.

Question 4

4. Cross-validation

Answer 4

1. Aknowledge that no true out-of-sample data exists

( in Trading only out-of-sample data is live trading data )

2. Iterative out-of-sample == overfitting

Question 5

5. Model dynamics

Answer 5

1. Beware of structural changes over time
2. NO tweaking –> overfitting

Question 6

6. Model Complexity

Answer 6

1. Beware of dimensionality curse
   ( more predictor variables == more data needed )
2. Aim for simplicity and Regularization and interpretability ( ML application should not be a black box )

Question 7

7. Research Culture

Answer 7

1. Establish a Research Culture That Rewards Quality Science
2. Be Careful with Delegated Research (assistants have incentive to support supervisor hypothesis)

Question 8

Protocol Summary

Answer 8

1. Motivation
2. Multiple Testing and statistic methods
3. Sample choice and data
4. Cross-validation
5. Model dynamics
6. Model Complexity
7. Research Culture