Reinforcement Learning

Question 1

What is reinforcement learning?

Answer 1

Operant conditioning

An agent interacting with its environment by building up an internal model through trial and error

Question 2

Components of an RL agent?

Answer 2

State
Transition
Action
Reward

Question 3

Whats the difference between deterministic and stochastic actions?

Answer 3

Deterministic - choice action will always occur

Stochastic - choice action has a chance to occur where probability represents uncertainty

Question 4

How do we represent solutions?

Answer 4

Using a policy: mapping from states to actions

Question 5

How do we evaluate policies?

Answer 5

Deterministic: Sum total rewards for following a policy
Stochastic: sum expected rewards for following a policy

Question 6

What are the two RL algorithm types? Describe them.

Answer 6

1. Model based - agent knows STAR

2. Model free - agent does not know T and R and must learn them through trial and error

Question 7

Explain model based RL.

Answer 7

Agent is given:
All STATES in the environment
Set of all ACTIONS in each state
TRANSITION probabilities between s and s' given a
REWARD for each action in each state

Question 8

Explain model free RL.

Answer 8

Model tries different actions in different states to build an estimate of TRANSITION probabilities and REWARDS for performing actions.

Question 9

Exploration vs Exploitation

Answer 9

Exploration: Finds more info about environment
Exploitation: Uses known info to maximise reward

Question 10

Passive vs Active RL

Answer 10

Passive: agent executes a fixed policy then evaluates it
Active: agent updates a policy as it learns

Question 11

Fully vs Partially Observable Environments

Answer 11

Fully: agent is initialised with state information and reward transition functions - knows current state, actions to transition to next state and reward for doing so

Partially: Agent has an internal model of the environment that it refines through trial and error where it can better learn states and transition functions

Question 12

What is a Markov Decision Process

Answer 12

A model of an environment that consists of:

1. finite # of states
2. probabilistic transitions between states
3. possible actions at a state
4. rewards for performing a specific action in a specific state

Question 13

What is the Markov Property?

Answer 13

A Markov process is a stochastic process who’s future state only depends on the current state and current action – not on past states/actions

Future is independent of the past given the present

Question 14

What are the 4 types of Markov Models

Answer 14

1. MDP - control over state transitions, fully observable
2. POMDP - control over state transition, partially observable
3. Markov Chain
4. HMM

Note that MDP and POMDP describe deterministic actions. The other 2 are the stochastic versions of the first

Question 15

What is the purpose of Gamma?

Answer 15

Discount rewards
We can control how much the agent cares about the future
Gamma close to 1 => agent cares a lot about the distant future
Hyper parameter

Question 16

What are some of the advantages of value iteration?

Answer 16

Will converge

Good for small number of states

Question 17

What are some disadvantages of value iteration?

Answer 17

Slow to converge for many states

Knowing T and R is unrealistic

Question 18

How do you perform value iteration?

Answer 18

Initialise the value of each state to zero and iteratively refine V(s) at each time step as it converges to V*(S)

Stope when the difference between the values of 2 consecutive iterations is below some threshold theta

Question 19

What is Direct Evaluation?

Answer 19

Model free RL

Every time you visit a state write down what the sum of rewards turn out to be then average those

Does this for every state to estimate T and R

can then start exploiting

Question 20

How do you perform direct evaluation?

Answer 20

1. Choose a starting state
2. Choose an action
3. Set every state traversed to -1 to ensure no backtracking
4. Average the sum of each value in a policy associated with a particular starting state every terminal state
5. Repeat for all states

That value will be the estimated R of starting state S

Question 21

What are some advantages of direct evaluation?

Answer 21

Easy to understand
Do not need T and R
Eventually computes correct state values

Question 22

What are some of the disadvantages of direct evaluation?

Answer 22

Takes long

Each state learned separately

Question 23

What is policy iteration?

Answer 23

Model - based learning

Initialise, Evaluate, Improve

Question 24

How to perform policy iteration?

Answer 24

Initialise an arbitrary policy - evaluate it using bellman’s (value iteration) - check if it can be improved by checking that the best action for every state is the one we currently have in our policy

Question 25

What are some advantages of policy iteration?

Answer 25

Converges on an optimal policy

Question 26

What are some disadvantages of policy iteration?

Answer 26

Slow.

Question 27

What is Q learning?

Answer 27

Model Free RL
POMDP
In every iteration:
1. agent observes state s
2. takes action a
3. observes new state s'
4. receives reward r

Question 28

What is the basis of Q learning? Explain this concept.

Answer 28

Temporal credit assignment. Agent takes a path that maximises total future reward i.e. the agent will not follow a greedy policy - balances current and future rewards.

Question 29

What is temporal difference learning?

Answer 29

Basis of Q learning

Adjust the estimated value of a state based on immediate reward and the estimated value of the next state

Question 30

How do you Q learn?

Answer 30

1. Init q table = 0
2. observe current state s
3. loop
   a. select and action
   b. receive immediate reward and observe new state s’
   c. update Q table using formula.

Question 31

What are some Q learning properties?

Answer 31

Converges if:
Enough exploration
Exploits over time
learning rate between 0 and 1

Please read over methods of changing learning rate

Question 32

What does learning rate do? Q learning

Answer 32

Impacts how drastically Q value of current state changes

Question 33

How do we choose actions in Q learning?

Answer 33

Greedy Epsilon approach
Epsilon determines the likelihood we take a random action
Over time e must decrease

Question 34

What are the three types of learning in multi-agent systems?

Answer 34

Cooperation
Competition
Individual