Lecture 2 - Machine Learning

Question 1

Learning

Answer 1

Acquiring new knowledge or skills and improve one’s performance

Question 2

A robot can learn about .. (2)

Answer 2

1. Itself: sensors or actuator info that might vary over time
2. Its environment: Learning maps, how to achieve goal

Question 3

(3) benefits of learning

Answer 3

1. Enabling to perform its task better
2. Adapting to changes in environment (hard to preprogram)
3. Simplify designer’s programming work

Question 4

3 Forms of Learning

1. Supervised learning

Answer 4

With external supervisor/teacher

In- and output pairs are presented & the function between these pairs is learned

Question 5

3 Forms of Learning

2. Unsupervised learning

Answer 5

All information should be taken from the inputs alone

-> it can be useful to preprocess inputs, e.g. divide in meaningful clusters

Question 6

3 Forms of Learning

3. Reinforcement learning

Answer 6

With an evaluation signal.

Question 7

What is feedback in supervised learning?

Answer 7

The target action or output for a specific input.

Question 8

Example of supervised learning + name

Answer 8

Neural network learning

-> the weights of the connections between nods are learned (connectionist learning)

Question 9

How did ALVINN learn to drive?

Answer 9

ALVINN steers how it think it should
Humans show how it should steer
ALVINN computes the error
ALVINN uses this error to update weights
-> REPEAT

Question 10

Disadvantage of supervised learning (3)

Answer 10

1. A trainer is needed (less autonomous)
2. Not online (first training then operating phase)
3. Not incremental (when it is operating, it is not learning)

Question 11

4 Characteristics of Reinforcement Learning

Answer 11

1. Learning from interaction
2. Goal-oriented learning
3. Learning about, from, and while interacting with an external environment.
4. Learning what to do (how to map situations to actions) so as to maximize a numerical reward signal

Question 12

5 Key features of Reinforcement Learning

Answer 12

1. Learner is not told which actions to take
2. Trial and Error search
3. Possibility of delayed reward
4. Sacrifice short-term gains for greater long-term gains
5. The need to explore and exploit
6. Considers the whole problem of a goal-directed agent interacting with an uncertain environment

Question 13

4 Characteristics of a complete agent

Answer 13

1. It is temporally situated
2. It learns and plans continually
3. The object is to affect the environment
4. The environment is stochastic / uncertain

Question 14

4 Elements of Reinforcement Learning (inwards to outwards)

1. Policy
2. Reward
3. Value
4. Model

Answer 14

1. Policy: What to do?
2. Reward: What is good?
3. Value: What is good because it predicts reward?
4. What follows what?

Question 15

Actuator space

Answer 15

Set of all possible actions

Question 16

When the robot knows/learned which action to perform in each state it has learned a …

Answer 16

Reactive controller

Question 17

Exploration (RL)

Answer 17

In order to learn the optimal action, the robot has to try everything (trial and error)

Question 18

Exploitation (RL)

Answer 18

Simultaneously to exploration, the robot should perform well and exploit what it has learned.

Question 19

Once mapping between inputs and actions is learned, the robot can just exploit the learned knowledge and stop exploring.. right?

Answer 19

No, not always. There might be (1) sensor errors (uncertainty) and (2) a changing environment.

Question 20

Exploitation/Exploration dilemma (Trade off between..(2))

Answer 20

1. Constantly learning - Exploration (possibly doing things less perfectly)
2. Constantly using what it knows (Exploitation) (Can not improve predictions of other actions)

Question 21

What is learned in RL? Consider robot’s actuator and sensor space!

Answer 21

The robot learns a value-function (possibly in table) with all possible state-action pairs along with their Q-values.

Question 22

Q-value

Answer 22

Grows if good things happen and shrinks if bad things happen.

Question 23

When is the RL table learning method efficient (2)?

Answer 23

1. When the state space is not too big

2. States and actions are discrete

Question 24

What if table learning method alone is inefficient?

Answer 24

Combine RL with function approximators such as neural networks

Question 25

Temporal Credit Assignment (RL)

Answer 25

In a maze, result of tested state-action pair may come long after the action
-> Rewards and punishment have to be backpropagated and given multiple previous state-action pairs.

Question 26

Notable RL Applications (2)

Answer 26

1. TD-Gammon: World’s best backgammon program

2. Elevator control

Question 27

Learning by Imitation
What does it free from?
What does it involve?

Answer 27

It frees from trial and error but is not trivial!

It involves careful decisions about internal representations

Question 28

Learning from Demonstration

Answer 28

Learning by experiencing a task directly (human controls the robot to let it experience the result of good actions)

Question 29

What does the robot need to learn in imitation/demonstration learning (2)?

Answer 29

1. What it experienced during trying

2. How it can generate that behavior again

Question 30

Why is forgetting important (2)?

Answer 30

1. Making room for new information

2. Replacing old information that is no longer correct

Question 31

What determines which type of learning methods are possible for a particular learning problem?

Answer 31

Amount and type of information (feedback, reward, punishment, error)

Question 32

Can a robot use multiple learning methods at the same time?

Answer 32

YES!