Class 4 - Adaptation

Question 1

3 types of adaptive processes

Answer 1

1. Evolutionary algorithms
2. Reinforcement learning
3. Learning by demonstration

Question 2

Evolutionary robots…

Answer 2

1. are created to adapt to their environment via evolutionary computing
2. can generate off-springs
3. mainly focus on evolving the brain
4. can work on adjusting issues by themselves

Question 3

Genotypes

Answer 3

numbers that describe the phenotype of the evolutionary robot

Question 4

Genotypes can be…:

A. discrete numbers between [0, 1]
B. continuous numbers between [0, 1]
C. both A and B
D. neither A nor B

Answer 4

C. both A and B

Question 5

T/F: Different parts of the genotype can describe different parts of the phenotype of the robot

Answer 5

True

Question 6

An evolutionary algorithm can be split into two main phases…

Answer 6

1. the “testing phase” –> where the robot is put in the environment and the fitness function is evaluated
2. the “generation phase”, where the resulting fitness function is used to create the next generation of offsprings

Question 7

3 methods to create off-springs (in the context of evolutionary algorithms)…

Answer 7

1. genetic algorithm
2. evolutionary strategy
3. modern evolutionary strategy

Question 8

Match the off-spring method (in the context of evolutionary algorithms) to its description…

A. genetic algorithm
B. evolutionary strategy
C. modern evolutionary strategy

1. only allows continuous values in the genotype which are then crossed-over with mutation (which allows a number between 0 and 1) to create the genotype of the off-spring. The first generation of genotypes can be random and parents can outlive the children if they have a better fitness function.
2. only allows discrete values in the genotype which are then crossed-over with (flipped value) mutation to create the genotype of the off-spring. Also, the first generation of genotypes can be random.
3. Uses correlation to see whether the previous children were performing better than the new version. Only uses one parent since it is computationally challenging.

Answer 8

A-2
B-1
C-3

Question 9

Advantages of evolutionary algorithms

Answer 9

If one component is weaker, the others can compensate!

Question 10

Disadvantages of evolutionary algorithms

Answer 10

1. unbounded complexity does not happen
2. adaptations are short-term → need to fix problems immediately
3. prevents exploring new ideas because offspring comes from a certain family

Question 11

T/F: in RL, only the off-spring “carries” the improvement

Answer 11

False, that’s the case generally for evolutionary algorithms. In RL, the robot improves itself continuously given a reward

Question 12

Policy

Answer 12

explains to the robot which action to take in a certain state

Question 13

Is the policy “ideal” before the robot starts exploring?

Answer 13

No, the robot updates the policy as it explores the environment

Question 14

Two types of policies

Answer 14

1. deterministic

2. stochastic

Question 15

Gaussian policy is an example of…

A. deterministic policy
B. stochastic policy

Answer 15

B. stochastic policy

Question 16

In a deterministic policy…

Answer 16

the robot has n categories that it can use / pick from

Question 17

In a stochastic policy…

Answer 17

there is a condition on execution: when the robot is in one given field, there is i.e., 6 % chance that the robot will move to the right, 2 % it will move to the left, 1% it will move up and 1% it will move down.

Question 18

T/F: A gaussian policy is used when the action space is continuous

Answer 18

True

Question 19

Finite-horizon undiscounted return

Answer 19

summing all the rewards and picking the policy which corresponds to the highest sum

Question 20

Infinite-horizon discounted reward

Answer 20

summing all rewards and subtracting all steps taken. if the result is negative, it means that we took more steps than the total reward, which is bad; if the result is positive, it means that we got a high total result with little number of steps, which is what we want

Question 21

Q-learning

Answer 21

1. the robot explores environment itself
2. Q-learning is an off policy reinforcement learning algorithm that seeks to find the best action to take given the current state. It’s considered off-policy because the q-learning function learns from actions that are outside the current policy, like taking random actions, and therefore a policy isn’t needed.

Question 22

Monte-carlo approach

Answer 22

class of computational algorithms that rely on repeated random sampling to obtain numerical results.

Question 23

T/F: the “learning by demonstration” adaptation method has 3 main phases.

If TRUE, name them
If FALSE, give the exact number

Answer 23

True;

demonstration phase; training phase; testing phase

Question 24

In the context of learning by demonstration > > in the demonstration phase…

…Match the following types of demonstrations with their description

A. kinesthetic teaching
B. tele-operated teaching
C. direct imitation of human behavior

1. imitate human’s behavior from observational human data
2. teacher uses controller to show the robot the correct behavior
3. teacher places the robot’s body in the correct position

Answer 24

C-1
A-3
B-2

Question 25

In the context of learning by demonstration, in which phase (demonstration / training / testing) does the robot actually learn from the observations?

Answer 25

Training phase