C8

Question 1

what is hierarchical reinforcement learning?

Answer 1

the granularity of abstractions is larger than the fine grain of the primitive actions of the environment (taking a train instead of individual steps)

Question 2

advantages of hierarchical methods

Answer 2

1. simplify problems through abstraction. Agent creates subgoals and solves these fine grain tasks first. Actions are abstracted into macro actions to solve these subtasks
2. increased sample efficiency: subpolicies are learned to solve subtasks, reducing the environment interactions. Subtasks can be transferred to other problems
3. policies become more general, and are able to adapt to changes in the environment more easily
4. the higher level of abstraction allows agents to solve larger, more complex problems

Question 3

disadvantages of hierarchical methods

Answer 3

1. Many assume that domain knowledge is available to subdivide the environment so that hierarchical RL can be applied
2. algorithmic complexity: identify subgoals, learn subpolicies etc.
3. macros are combinations of actions and the number of combinations of actions is exponential in their length, so computational complexity of the planning and learning choices increases by the introduction of the macro actions
4. the quality of a behavioral policy that includes macro-actions may be less than that of a policy consisting only of primitive actions, because they may skip over possible shorter routes, that the primitive actions would have found

Question 4

what is the options framework?

Answer 4

Whenever a state is reached that is a subgoal, then, apart from following a primitive action (main policy), you can follow the option policy, a macro action consisting of a different subpolicy specially aimed at satisfying the subgoal in one large step. In this way macros are incorporated into the reinforcement learning framework.

Question 5

what is an option?

Answer 5

a group of actions with a termination condition

Question 6

what is an option?

Answer 6

a group of actions with a termination condition. They take in environment observations and output actions until a
termination condition is met.

Question 7

what are the tree elements of an option 𝜔?

Answer 7

1. initialization set I_𝜔: the states that the option can start from
2. subpolicy 𝜋𝜔 (𝑎|𝑠): internal to this particular option
3. terminal condition 𝛽𝜔 (𝑠): tells us if 𝜔 terminates in s

Question 8

what are macros?

Answer 8

any group of actions, possibly open-ended

Question 9

what is intrinsic motivation?

Answer 9

An inner drive to explore, named so to contrast it with classic extrinsic motivation (the conventional RL reward signal). They are related to reward signals for achieving subgoals

Question 10

How do multi agent and hierarchical reinforcement learning fit together?

Answer 10

agents often work together in teams or other hierarchical structure

Question 11

what is so special about Montezuma’s Revenge?

Answer 11

it is a difficult situation to learn for RL, because is has little reward signal and the reward signal is delayed. It consists of long stretches in which the agent has to walk without the reward changing.