Final Review pt. 6

Question 1

True or False: Options over an MDP form another MDP. Why?

Answer 1

True. Options over MDPs form SMDPs which can be converted to MDPs.

Question 2

Can we do Q learning on SMDP? If so, how do we do that?

Answer 2

True, we use observations in place of actions and discounted rewards in place of rewards

Question 3

Is value iteration guaranteed to converge to the optimal policy for an SMDP?

Answer 3

with a particular choice of option there is no guarantee that it would end up with
an optimal policy

Question 4

Is value iteration guaranteed to converge for an SMDP?

Answer 4

I’m thinking value iteration is guaranteed to converge with an SMDP but not necessarily to the optimal policy.

Question 5

Does SMDP generalize over MDP?

Answer 5

Yes, SMDP is a generalization of MDP. In SMDP, the timestep between actions is a variable.

Question 6

How is SMDP defined? Why is it called semi-Markov?

Answer 6

SMDPs are Markov Decision Processes that use options instead of discrete “atomic” actions. These options are allowed to take variable time instead of discrete time.

Question 7

How is an option defined?

Answer 7

<i> : I = the initialization set of states, Pi = the policy to take with that option, and Beta = the termination set of states.</i>

</i>