10 - Policy Optimization

Question 1

Value Function Approximation (VFA) goes from tabular to parameterized value estimates. Policy Search goes from _____________ to _____________ policy estimates.

Answer 1

Value Function Approximation (VFA) goes from tabular to parameterized value estimates. Policy Search goes from tabular to parameterized policy estimates.

Question 2

Policy search typically results in __________ {low, high} variance estimates of policy.

Answer 2

Policy search typically results in high variance estimates of policy.

Question 3

In general, one of the biggest reasons to choose gradient-free optimization, over gradient-based optimization, is the error function is not ____________.

Answer 3

In general, one of the biggest reasons to choose gradient-free optimization, over gradient-based optimization, is the error function is not continuous (and convex).

Question 4

One way to improve policy search is to use domain knowledge to ____________ the space of possible policies. (from ~26:25)

Answer 4

One way to improve policy search is to use domain knowledge to decrease the space of possible policies. (from ~26:25)

Question 5

In order to analytically calculate the gradient, the policy has to be ____________ whenever it is non-zero.

Answer 5

In order to analytically calculate the gradient, the policy has to be defined (or be numerical) whenever it is non-zero.

Question 6

What are the 3 steps in all RL algorithms?

1. __________ the policy
2. __________ the policy
3. __________ the policy

Answer 6

1. Run the policy: actually act in env
2. Evaluate the policy: estimate V𝜋 or Q*
3. Improve the policy: do something which lets you pick better actions

Question 7

Briefly, what is the log-derivative trick?

Answer 7

It is a way of deriving the policy gradient by taking the derivative of log probability, which gives us the score function and has the nice property that its expectation is 0.

Question 8

Briefly, what is the advantage function?

Answer 8

This is the difference between the Q value of a given state-action pair and the value function of that state.

Question 9

Policy gradient methods are ______________ {off policy, on policy}.

Answer 9

Policy gradient methods are on policy.

Question 10

The estimated value of policy is typically a ___________ {regression, classification} supervised machine learning problem.

Answer 10

The estimated value of policy is typically a regression supervised machine learning problem.

Question 11

The bias-variance trade-off for policy gradient methods can be managed with _____________ .

Answer 11

The bias-variance trade-off for policy gradient methods can be managed with choice of n (our steps).