Reinforcement Learning

Question 1

what is reinforcement learning?

Answer 1

an agent acts in an environment and is rewarded depending on its actions

Question 2

stochastic environment

Answer 2

no guarantee of same outcome if taking the same action in the same state

Question 3

what is the agent given in rl?

Answer 3

initial state
set of possible actions
reward associated with each state
not given transition model

Question 4

state

Answer 4

state of the environment as perceived by the agent

Question 5

policy

Answer 5

strategy that defines the behavior of an agent by mapping states to actions

guides the agent on what actions to take in each state to maximize cumulative rewards over time

“decision-making function” of the agent

Question 6

reward

Answer 6

feedback signals that an agent received from environment after taking an action
quantified value

Question 7

utility

Answer 7

subjective value agent places on being in a given state capturing both immediate rewards and future (discounted) rewards

Question 8

discount factor

Answer 8

value between 0 and 1 to determine how importance placed on future rewards
0 being only care about immediate rewards
closer to 1 agent considers future rewards almost as important as current reward

Question 9

what is the utility of the terminal state?

Answer 9

just reward of that state

Question 10

utility of a state formula

Answer 10

U(st)= rst + yU(st+1)

Question 11

q-value

Answer 11

expected utility of taking a particular action in a state

Question 12

q table

Answer 12

table of q values for all possible combinations of actions and states

Question 13

q table is used as the agents

Answer 13

policy

Question 14

how is the q table used to make an action?

Answer 14

chose action with largest q value and this max expected reward over time

Question 15

softmax

Answer 15

a row of q table can be transformed into a probability distribution of actions using softmax function

Question 16

agent uses softmax to

Answer 16

balance exploration and exploitation

Question 17

softmax is alternative to just selecting

Answer 17

action with highest q value

Question 18

Temperature parameter

Answer 18

T>0
higher encourages exploration by making action probability more even
lower encourages exploitation as action probability favours those with higher q values

Question 19

exploration

Answer 19

agent needs to explore environment to find out whether a better strategy exists

Question 20

exploitation

Answer 20

agent needs to exploit what it has already learnt about environment to make good choices
leverage agent’s current knowledge

Question 21

temporal difference learning

Answer 21

unsupervised learning technique used in reinforcement learning for the purpose of predicting the total reward expected over future

Question 22

4 steps for temporal difference learning

Answer 22

1) chose T, discount rate and desired learning rate
2) set all values in q table to 0
3) for each episode
start at initial state
chosen action using policy
take action and observe r and next state
update q table for state action pair using temporal difference update rule (based on difference between predicted reward and actual reward= TD error)
4) repeat for multiple episodes until q value converges