Rewards

Question 1

Why would we want to change the reward function for an MDP?

Answer 1

To make the MDP easier (speed, space, solvability) to solve while learning something similar to what it would have learned anyways

Question 2

How can we change the reward function without changing the optimal policy?

Answer 2

1. Multiplying by a (positive) scalar
2. Shifting by a scalar (adding)
3. Non-linear potential-based transformations

Question 3

What is the new Q function equal to if we multiply the reward function by a positive constant c?

Answer 3

Q’(s,a) = c*Q(s,a)

Question 4

What is the new Q function equal to if we add a constant c to the reward function?

Answer 4

Q(s,a) = Q(s,a) + c/(1-gamma)

Question 5

What is potential based reward shaping? What is the purpose?

Answer 5

Adding rewards for entering states but subtracting them when the state is exited. It is intended to encourage specific behavior (e.g. moving towards a goal) and speed up learning without creating an infinite reward pump.

Question 6

What is equivalent to doing Q learning with potentials?

Answer 6

Q learning initialized with the potential function