Reinforcement Learning

Question 1

What is a trajectory?

Answer 1

A trajectory τ is a sequence of states and actions in an environment.

τ = (s₀, a₀, s₁, a₂, …)

Question 2

What are other names for a trajectory?

Answer 2

An episode or a rollout.

Question 3

What is a reward function?

Answer 3

The reward function of an envrionment measures how good state-action pairs are:

r_s = R(s_t,a_t)

Question 4

What is the return of a trajectory?

Answer 4

The return is the measure of cumulative reward along it.

Question 5

What is the finite horizon undiscounted sum method of calculating return?

Answer 5

Finite horizon undiscounted sum of rewards:

Question 6

What is the infinite horizon discounted sum method of calculating return?

Answer 6

Infinite horizon discounted sum of rewards:

Question 7

What is a policy?

Answer 7

A policy π is a rule for selecting actions. It can be stochastic or deterministic.

Question 8

What is a stochastic policy?

Answer 8

A stochastic policy gives a probability distribution over actions, and actions are selected randomly based on that distribution.

Question 9

What is a deterministic policy?

Answer 9

A deterministic policy maps π directly to an action.

Question 10

What is the goal of reinforcement learning?

Answer 10

To learn a policy which maximizes expected return.

Question 11

What is the optimal policy π*?

Answer 11

The optimal policy π* is:

Question 12

What are the two main approaches for solving the optimal policy problem?

Answer 12

1. Policy Optimization
2. Q-Learning