C7

Question 1

Why is there so much interest in multi-agent reinforcement learning?

Answer 1

multi-agent is much more realistic because in the real world we also have multiple agents

Question 2

What are the 3 main challenges of multi-agent reinforcement learning?

Answer 2

1. partial observability
2. nonstationary environments
3. large state space

Question 3

what is a Nash strategy?

Answer 3

when the agent is guaranteed to do no worse than tie against any other opponent strategy

Question 4

what is the Nash equilibrium?

Answer 4

a situation where no agent has anything to gain by changing its own strategy (minimax). The agent does not try to exploit the opponent strategy’s flaws, it just wins when the opponent makes mistakes

Question 5

what is a Pareto Optimum?

Answer 5

the best possible outcome for us where we do not hurt others, and others do not hurt us. It is a cooperative strategy.

Question 6

what is the Pareto efficient solution?

Answer 6

the situation where no cooperative agent can be better off without making at least one other agent worse off

Question 7

in a competitive multi-agent system, what algorithm can be used to calculate a Nash strategy?

Answer 7

Counterfactual Regret Minimization (CFR)

Question 8

What makes it diffcult to calculate the solution for a game of imperfect information?

Answer 8

it increases the size of the state space, and computing the unknown outcomes quickly becomes unfeasible

Question 9

what is the Prisoner’s dilemma?

Answer 9

• if both prisoners confess they both get 5 years in prison
• if both stay silent they both get 2 years in prison
• if I confess and the other stays silent, I walk free
• if I stay silent and the other confesses, I get 10 years

this is an example of mixed behaviour

Question 10

what are the Pareto and the Nash strategies in the Prisoner’s dilemma?

Answer 10

Pareto: both stay silent (cooperate)
Nash: both confess (defect)

Question 11

what is the Iterated Prisoner’s dilemma?

Answer 11

for multiple rounds of the Prisoner’s dilemma a tit for tat strategy works best: in the first round you play cooperative, after that you play whatever the opponent did in the previous round

Question 12

Name two multi-agent card games of imperfect information

Answer 12

poker, blackjack, bridge

Question 13

name three kinds of strategies that can occur in multi-agent reinforcement learning

Answer 13

• CFR
• evolutionary strategies
• cooperative strategies

Question 14

Counterfactual Regret Minimization

Answer 14

a statistical algorithm that converges to a Nash equilibrium. Unlike minimax, it is suitable for imperfect information games

Question 15

swarm computing

Answer 15

focuses on emerging behavior in decentralized, collective, self-organized systems. Introduces forms of communication between agents

cooperation and survival of the group (Pareto)

Question 16

Name two solution methods that are appropriate for solving mixed strategy games

Answer 16

evolutionary methods and cooperative methods

Question 17

evolutionary algorithms

Answer 17

inspired by bio-genetic processes of reproduction: mutation, recombination, selection. repeat:
1. Evaluate the fitness of each individual of the population
2. Select the fittest individuals for reproduction
3. Through crossover and mutation generate new individuals
4. Replace the least fit individuals by the new individuals

focus on competition and survival of the fittest (Nash)

Question 18

what is regret?

Answer 18

the regret of an action is the amount of reward that is missed by an agent for not choosing the actions with the highest payoff

Question 19

what 3 types of behaviour do we see in MARL?

Answer 19

cooperation, competition and mixed behaviour

Question 20

what is Game Theory?

Answer 20

the study of strategic interaction among rational decision-making agents

Question 21

Describe the behaviour that emerged in Hide and Seek

Answer 21

Collaboration

Question 22

what is population-based training?

Answer 22

it combines evolutionary ideas with RL ideas

teams compete against each other and if a team learns good behaviour they survive: team-based learning of better and better behaviour

Question 23

why is StarCraft used as a testbed or MARL?

Answer 23

to study population-based training: there is collaboration within teams and competition between teams