Module 9: Reinforcement Learning

Question 1

Which of the following is given to the agent when using passive reinforcement learning? Check all that apply.

The agent’s policy π.
All of the states that can be reached by the agent.
The transition model P(s’ | s, a).
The reward function R(s) that specifies the reward for each state.

Answer 1

The agent’s policy π.
All of the states that can be reached by the agent.

The goal of passive reinforcement learning is to compute values for each state under policy π, the transition and reward are not required for direction evaluation.

Question 2

T/F
Direct utility estimation computes a Utility function U by considering the connection between the utility of a state and the utility of its successor states.

Answer 2

False

Direct utility estimation only use the sample transitions.

Question 3

Which of the following is true of both adaptive dynamic programming (ADP) learning and temporal-difference (TD) learning?

Answer 3

The utility of a state is adjusted locally to agree with the utility of at least one successor state.

The utility of a state is adjusted locally to agree with the utility of at least one successor state.

Question 4

T/F
Q-learning with linear function approximation (weighted linear function of a set of features) will always converge to the optimal policy.

Answer 4

False

It is possible for Q learning with linear function approximation to converge to the optimal policy, but there is no guarantee that it always finds such an optimal policy. For example, consider the problem where the true Q-function is a quadratic function. Q(s, a) where a= 0 or 1 and s can be any real number and the true function is Q(s,0) = s^2 and Q(s,1) = 3s^4 for all s. Then if our function approximator is linear Q(s,a) = ks + la + b, then there exists no k,l, and b that can represent the true Q function because we are trying to fit a linear to a quadratic function.

Question 5

In reinforcement learning, a deterministic policy is

Answer 5

A mapping from states to actions

Question 6

Which of the following equations are updates for TD Q-learning? Check all that apply.

𝑄(𝑠,𝑎)⟵𝑄(𝑠,𝑎)+𝛼(𝑅(𝑠)+𝛾max𝑎1𝑄(𝑠1,𝑎1)−𝑄(𝑠,𝑎))
𝑄(𝑠,𝑎)⟵𝑄(𝑠,𝑎)+𝑎(𝑅(𝑠)+Υ𝑄(𝑠1,𝑎1)−𝑄(𝑠,𝑎)

Answer 6

𝑄(𝑠,𝑎)⟵𝑄(𝑠,𝑎)+𝛼(𝑅(𝑠)+𝛾max𝑎1𝑄(𝑠1,𝑎1)−𝑄(𝑠,𝑎))