8 - Deep Reinforcement Learning

Question 1

Reinforcement Learning

Answer 1

Solve via trial and error to maximise rewards

Question 1

Deep q learnign

Answer 1

Use a neural network (CNN) to predict the Q values and then select the biggest Q value

Question 2

Deep Q Learning Network Architecture

Answer 2

3 conv layers and 2 fully connected

Question 3

How to fix: Consecutive samples might be correlated (Deep Q Learning)

Answer 3

Store the agent’s experiences and randomly create mini-batches from the pool of stored samples.

(Gives a variety of data)

Question 4

How to fix: Small updates to Q value may significantly change the policy

Answer 4

Update the network weights every 10000 steps rather than each individual step. (update target model every 10k steps)

Question 5

Epsilon value

Answer 5

Starts at 1.

Refers to the probability of choosing to explore.

Question 6

Exploration

Answer 6

Eg select a random action

Allows an agent to improve its current knowledge

Question 7

Exploitation

Answer 7

Behave as the robot has learnt so far.

Choose the greedy action to get the most reward by exploiting the agent’s current action-value estimates. May be sub-optimal

Question 8

Epsilon-Greedy Action Selection

Answer 8

With probability epsilon ε, select random action a,
otherwise select at = argmaxaQ(Φ(st),a;θ)

balance exploration/exploitation

Question 9

Store transition

Answer 9

Store state to solve continuous frame problem.

Question 10

Sample random minibatch of transitions

Answer 10

Use random function to select data from memory