CS7642_Week2

Question 1

How do we evaluate a learner?

Answer 1

1. Value of the returned policy
2. Computational complexity (time)
3. Experience complexity (i..e how much data it needs)

Question 2

What are the 3 “classes” of solution methods for solving RL problems? (bonus: what category do TD methods fall into?)

Answer 2

1. Model-based
2. Value-based (TD methods fall into this)
3. Policy-based

Question 3

What properties must the learning rate have for RL?

Answer 3

1. SUM of all learning rate values must be infinite

2. SUM OF SQUARES of learning rate values must be finite

Question 4

Names some of the differences between TD(0) and TD(1)

Answer 4

TD(0):

• Slow to propagate information
• High bias, low variance
• Maximum likelihood estimate (MLE)

TD(1):

• Equivalent to MC, samples full trajectories
• Requires full trajectory in order to update
• Low bias, high variance

Question 5

What values lambda tend to work well (empirically speaking) when used in TD(lambda)?

Answer 5

0.3-0.7

Question 6

Does Q-learning always converge? If so, what does it converge to?

Answer 6

Yes, it does converge, and in fact converges to the optimal value Q*

Question 7

What is non-expansion/contractions?

Answer 7

TODO: Watch lesson 5 on convergence (need to particularly pay attention to stuff on contractions and non-expansion at a conceptual level)

Question 8

What things are contraction mappings / non-expansions?

Answer 8

Order statistics, FIXED convex combinations