Introduction

Question 1

States, actions, and rewards - formal definition

Answer 1

Question 2

General transition function

Answer 2

Question 3

Total Return

Answer 3

Question 4

The objective function is the expected return

Answer 4

Question 5

An action sampled from a policy is written as

Answer 5

Question 6

Trajectory notation

Answer 6

Question 7

Return of a trajectory

Answer 7

Question 8

Expected return over all completed trajectory

Answer 8

Question 9

The policy gradient solves for

Answer 9

Question 10

Maximization with gradient

Answer 10

Question 11

The term π_θ(a_t | s_t) is the probability of the action taken by the agent at time step t. The action is sampled from the policy, a_t ∼ π_θ(s_t). The right-hand side of the equation states that the gradient of the log probability of the action with respect to θ is multiplied by return R_t(τ ).

Answer 11

Question 12

Probability of the entire trajectory

Answer 12