Hypothesis Testing in AI

Question 1

What are the 4 steps in hypothesis testing?

Answer 1

1. State the hypothesis
2. Design the statisitcal test
3. Calculate the test statistic
4. Reject or fail to reject the null hypothesis

State, Design,Calcuate, R or A

Question 2

What is a test statstic?

Answer 2

It is a function of the observed values of the random variables of interest

Question 3

What is meant by the p-value?

Answer 3

The probability in the tail

The probabability of observing a sample estimate as extreme as the one observed, assuming the null hypothesis is true.

Question 4

What is meant by the level of significance?

Answer 4

The probability of making a Type I error

Question 5

What is Type I error?

What is Type II error?

Answer 5

There are 2 types of errors,

Type I: Wrongly rejecting a true null hypothesis

Type II; Accept (Fail to reject) an unture null hypothesis.

The probability of a Type I error is the level of significance.

Question 6

What are the reasons for unrepresentative datasets?

Answer 6

1. Selection bias (leaving certain obervations out);
2. Survivorship bias (leave out funds no longer in existence);
3. Self selection bias (Fund Managers leave out certain funds)

The 3 S’s

Question 7

Data mining

Data dredging (Data snooping)

Answer 7

Data mining; vigorously testing until valid relationships are established

Data dredging: Overuse of statistical tests or Running hundreds of statistical tests to identify significant relationship without regard to economic rationale.

Question 8

Backtesting, overfitting and Backfilling

Answer 8

• Backtesting: Apply models on historical data to determine how well the model would have explained the actuals results: Good sometimes
• • Overfitting;When many parameters are used to a model in histotical data;Not good
• • Backfilling: Updating databases by inserting returns that pre-date the date of entry in the database. Backfill bias is also known as instant history bias. Generally not good

Question 9

Cherry picking and chumming

Answer 9

• Cherry picking ; Selectivley reporting results;
• Chumming: If enough predictions are made, some will be correct

Question 10

Why are cumulative return charts deceptive?

How is deception avoided?

Answer 10

Because of compounding effects. Gap widens with an early period advantage.

Deception is avoided by using cumulative log chart becase log returns are additive, not multiplicative.