CS7642_Week3

Question 1

What is the difference between RL with control and RL without control?

Answer 1

Control in RL just means that actions are being chosen by the learner.

Question 2

What does a contraction mapping do when applied to to functions F and G?

Answer 2

It brings them closer together

Question 3

What three things need to hold for the convergence theorem to guarantee that Q converges in the limit?

Answer 3

1. Averaging over noisy transitions
2. Update operator must be a non-expansion (i.e. a contraction. This is calculation that is essentially the one-step lookahead that we did in value iteration
3. Learning rate sequence must sum to infinity, sum of squares must be finite.

Question 4

How might we compute how much we do (or don’t) care about future rewards?

Answer 4

It’s proportional to something like H ~ 1 / (1 - gamma)

Question 5

Why is it not a good idea to set gamma to very small values?

Answer 5

You end up with an agent that acts myopically, always seeking immediate gratification rather than playing “the long game”

Question 6

What are three important features of policy iteration (PI)? What is the tradeoff we make when using PI over VI? What is the most important feature of PI?

Answer 6

1. Q –> Q* in the limit
2. Convergence is exact and complete in finite time (assuming use of the greedy policy)
3. Computes at least a fast as value iteration (VI)

The downside of PI is computational complexity because now we have to iterate over a full policy space rather than just at states / state-actions

Most important thing about PI is that it can’t get stuck in a local minimum (because of value improvement/non-improvement).

Question 7

What is bounded loss/regret with respect to policies?

Answer 7

A policy that epsilon-optimal, i.e. that at each timestep produces a value that is no further than epsilon away than what would have been achieved by the optimal policy.