Class 6 - Guest lecture - Deep Reinforcement learning

Question 1

Difference between reinforcement learning deep learning

Answer 1

The difference between them is that deep learning is learning from a training set and then applying that learning to a new data set, while reinforcement learning is dynamically learning by adjusting actions based in continuous feedback to maximize a reward.

Question 2

Traditional methods for robotics

Answer 2

work well for demos and narrow applications, but they don’t generalize well and require expensive and tedious adaptation to any new task or environment. (i.e., Boston robotics)

Question 3

Deep Learning for Robotics

Answer 3

has proven effective to achieve (super)humans-level performance on many tasks:

1. object detection and recognition of faces
2. speech recognition
3. dexterous manipulation
4. still in the very early stages

Question 4

How to apply DL to robotics options:

1. “easy” fix (what is it about)?
2. “harder” fix (what is it about)?

Answer 4

1. easy fix: replace some components with neural networks, BUT we still have to engineer the entire system, and design (and train) the different components separately (issues that may arise: mistakes in pipeline, not great general movements)
2. hard fix: end-to-end learning, automatic learning technique where the model learns all the steps between the initial input phase and the final output result. It takes the input and returns a distribution over action.

Question 5

The reinforcement approach to learning a solution…(pick one):

A. uses simulations to train the agent
B. places the agent in an environment and lets it explore this environment by performing actions which will cause a new state and reward for the agent.

Answer 5

B

Question 6

A solution to the fact that reinforcement learning generally requires a lot of time and a lot of repetitions is…

Answer 6

to use simulations to train thousands of agents in parallel

Question 7

Although deep learning has proven to be more robust against perturbations during training / testing, it still reports one main issue, that is…

Answer 7

even the best simulations are too different from reality

Question 8

reality gap (in the context of deep learning)

Answer 8

you might lose a lot when moving from simulation to reality

Question 9

Fill in:

In the context of deep learning simulations, a small error compound at each time step might result in very - similar / different - trajectories between simulation and real world.

Answer 9

different

Question 10

One approach that tries to solve the reality gap issue is the…

Answer 10

Sim2Real approach

Question 11

The Sim2Real approach uses dynamics randomization to…

Answer 11

train robots in simulation using a wide range of physics (e.g., amount of gravity, size of each component of the robot, frictions, visual appearance and lights, etc…) to force the robot to work over many different environments, with the hope that the real world ends up included.

dis: very computationally expensive

Question 12

One big problem in reinforcement learning is that (multiple picks are possible):

A. we have to design the reward function by hand
B. the reward function is always the same
C. the reward function we choose may not result in the behavior we want

Answer 12

A, C

Question 13

The idea behind imitation learning is to…

Answer 13

collect demonstrations from humans solving the target task (in the demonstration phase), and use them to train an agent (training phase + test phase).

Question 14

3 main approaches to imitation learning in the context of deep learning

Answer 14

1. behavior cloning
2. inverse reinforcement learning
3. sequence modelling

Question 15

In the context of imitation learning in deep learning, behavior cloning…

Answer 15

1. treats the problem as supervised learning.
2. Collects (state, action) pairs from many demonstration episodes.
3. Trains a neural network to produce the same actions on the same states.

Question 16

Main issues with behavior cloning

Answer 16

Since the agent tends to overfit the trajectories of the demonstrator, if the agent’s trajectory deviates from the
demonstrations, it can quickly diverge by compounding of
errors.

Question 17

Solution to the behavior cloning problem

Answer 17

Do not train an agent to memorize a trajectory path, but more to learn to assign a certain probability to a trajectory. This probability will tell how likely this trajectory was taken by a human. If the probability is high, then the action was likely taken by a human, hence the robot will follow that action.

Question 18

Incompliance mode

Answer 18

robot producing motor commands to its motor but also allows humans to move them.

Question 19

In the inverse reinforcement learning approach…

Answer 19

the agent infers the reward function that generated the behavior of another agent (from human demonstrations) and attempts to reproduce the same behavior using the inferred reward function

Question 20

Main issue with the inverse reinforcement learning approach

Answer 20

There are many reward functions that can explain the

same observed behavior, so how can we differentiate them?

Question 21

In the sequence modeling approach…

Answer 21

the goal is to predict a full sequence of actions that leads to a sequence of high rewards. Specifically, we want to learn the probability distribution over the most successful sequence of actions.

Question 22

Task specification

Answer 22

1. The reward function determines what behavior is learnt.

2. Different reward functions can make learning easier or harder on the same task.

Question 23

Reward hacking

Answer 23

occurs when an agent learns to exploit a poorly specified reward function to obtain high rewards by ‘cheating’.

Question 24

Example of reward hacking

Answer 24

a coffee-making robot is incentivized to learn all the steps to make a cup of coffee. One of the steps that is rewarded is ‘turn on the coffee machine’. A naive implementation of the reward function may lead to the robot repeatedly turning the machine on and off to keep collecting the same reward multiple times.

Question 25

A cooperation game shows NO nash equilibrium when…

Answer 25

one of the teams can improve its strategy without waiting for the other team to change its strategy

Question 26

A competition game shows NO nash equilibrium when…

Answer 26

one of the teams can improve its strategy without waiting for the other team to change its strategy