M04 - Adaptive & Learning in Autonomous Systems

Question 1

What learning methods do we have?

Answer 1

• reinforcement learning methods
• evolutionary methods

Question 2

What are the traits of reinforcement learning methods?

Answer 2

• a form of trial and error learning
• operates on a single individual
• varies the parameters of the robot on the basis of a reward on each step
• adapts only the connection weights of the neural network
• sample efficient
• complex & more subjected to instability

Question 3

What are the traits of evolutionary methods?

Answer 3

• operated on a population of individuals
• varies the parameters of the robot on the basis of a reward that performance of the on the entire evolution
• maximize cumulative reward
• slower than reinforcement learning

Question 4

What is adaptivity?

Answer 4

To develop the behavioral and cognitive skills required to perform a desired function

Question 5

What are the methods to achieve adaptivity?

Answer 5

• evolutionary methods
• learning from demonstration
• reinforcement learning

Question 6

Describe learning from demonstration.

Answer 6

• paradigm to enable robots to autonomously perform new tasks
• no need to analytically decompose and manually program desired behavior
  appropriate robot controller can be derived from observations of a human’s own performance

Question 7

What is the general approach of demonstration learning?

Answer 7

Demonstrate -> Train a model -> Evaluate
- a robot that can perform the desired task
- a task demonstrator that can effortlessly perform the task

Question 8

Why is learning from demonstration useful?

Answer 8

• formal descriptions some tasks are hard to define
• daily life activities are apparently easy for humans but computationally expensive for robots
• we do not have enough robotics to work on all activities

Question 9

What are the types of learning from demonstration?

Answer 9

• kinesthetic teaching
• teleoperation
• direct imitation of human behavior

Question 10

What is kinesthetic teaching?

Answer 10

The actuators of the robot are set in a passive mode and the experimenter physically manipulates the robot so to force it to produce the desired behavior

Question 11

What is teleoperation?

Answer 11

The actuators of the robot are controlled by the experiment through a joystick and/or a haptic device

Question 12

What is direct imitation of human behavior?

Answer 12

The training set is generated from the observation of a human displaying the desired behavior

Question 13

What are the categories of machine learning?

Answer 13

• supervised learning
• unsupervised learning
• reinforcement learning

Question 14

Describe, give an example and application of supervised learning.

Answer 14

• labeled data
• feedback
  e.g. Regression, SVM, Neural Network, etc.
  App. Object recognition

Question 15

Describe, give an example and app of unsupervised learning.

Answer 15

• no labeled data
• no feedback
  e.g. K-means, self-organizing maps
  App. Clustering

Question 16

Describe, give an example and application of reinforcement learning.

Answer 16

• reward-based learning
• increase cumulative discounted reward
  e.g. Q-learning, SARSA, TD- learning, DQNs
  App. Robot navigation

Question 17

What is the problem task and goal of reinforcement learning?

Answer 17

Problem: easy to evaluate but hard so solve
Goal: learn the action that maximizes (cumulative discounted) reward
E.g. outdoor navigation

Question 18

What tuple does the Markov decision process describe?

Answer 18

• S: set of possible states
• A: set of possible actions
• P: policy: state, action transition probability
• R: reward function for a (state, action) pair
• γ: discount factor to determine whether the current or future reward is valuable for the agent

Question 19

What is the Markov (memoryless) property?

Answer 19

The current state of the agent contains all necessary information about the world. The history of the state is not important to decide the next state.

(doesn’t need to remember previous states to make decision on future states)

Question 20

What is the Q update function?

Answer 20

• The learning rate determines to what extent newly acquired information overrides old information.
• the discount factor determines the importance of future rewards

Question 21

Where do rewards come from?

Answer 21

Reward is a signal from the environment

Question 22

What are the challenges in reinforcement learning?

Answer 22

• Credit assignment problem
  [which action leads more reward]
• High number of the states
  [The number of states may not fit in to the memory]
• Trade-off between exploration and exploitation
  [Should agent try new actions or exploit the “learned” one]
• Non differentiability of the real world
  [Noise in the environment, unexpected changes]
• The agent modifies the environment while learning
  [No batch mode for collecting data]

Question 23

How can we answer the challenges in reinforcement learning?

Answer 23

Utilize neural networks