10 Reinforcement Learning*

Question 1

what is q learning

Answer 1

create table storing state-action pairs

update table

Question 2

what is the pseudo code for q learning

Answer 2

initialise Q(s, a)
initialise Q(termical) to 0
for each episode:
initialise state S
for each step in episode:
do
A <- select action
take action A, then observe reward R and next state S
Q(S, A) <- Q(S, A, + a[R + ymax Q(S, a) - Q(S, A)]
update state
while S is not terminal
end
end

Question 3

how is q learning different

Answer 3

off policy by choosing the action with the max q value for the state

Question 4

how is sarsa different

Answer 4

on policy by choosing the action defined by the policy and updates its q value

Question 5

what is the pseudo code for sarsa

Answer 5

initialise Q arbitrarily
repeat (for each episode):
initialise s
choose a from s using policy derived from Q(eg. epsilon greedy)
repeat (for each step of episode):
take action a, observe r, s’
choose a’ from s’ using policy derived from Q
update s<- s’; a <- a’
until s is terminal

Question 6

what does RL solve task

Answer 6

1. use model based approach
2. value learning by analysing how good to reach a certain state or take specific action
3. derive a policy that maximize rewards (policy gradient)

Question 7

value iteration

Answer 7

1. start with a random value function
2. algorithm simpler
3. guaranteed to converge
4. more expensive
5. require more iterations

Question 8

policy iteration

Answer 8

1. start with a random policy
2. algorithm more complex
3. guaranteed to converge
4. cheaper to compute
5. require fewer iterations

Question 9

epsilon greedy

Answer 9

1. exploration
   choose action randomly
2. exploitation
   choose action based on highest rewards

Question 10

2 other models

Answer 10

1. actor critic
2. imitation learning