Muller, P;
Omidshafiei, S;
Rowland, M;
Tuyls, K;
Pérolat, J;
Liu, S;
Hennes, D;
... Munos, R; + view all
(2020)
A Generalized Training Approach for Multiagent Learning.
In:
Proceedings of the 8th International Conference on Learning Representations, ICLR 2020.
(pp. pp. 1-35).
ICLR
Preview |
Text
A Generalized Training Approach for Multiagent Learning.pdf - Published Version Download (1MB) | Preview |
Abstract
This paper investigates a population-based training regime based on game-theoretic principles called Policy-Spaced Response Oracles (PSRO). PSRO is general in the sense that it (1) encompasses well-known algorithms such as fictitious play and double oracle as special cases, and (2) in principle applies to general-sum, many-player games. Despite this, prior studies of PSRO have been focused on two-player zero-sum games, a regime where in Nash equilibria are tractably computable. In moving from two-player zero-sum games to more general settings, computation of Nash equilibria quickly becomes infeasible. Here, we extend the theoretical underpinnings of PSRO by considering an alternative solution concept, α-Rank, which is unique (thus faces no equilibrium selection issues, unlike Nash) and applies readily to general-sum, many-player settings. We establish convergence guarantees in several games classes, and identify links between Nash equilibria and α-Rank. We demonstrate the competitive performance of α-Rank-based PSRO against an exact Nash solver-based PSRO in 2-player Kuhn and Leduc Poker. We then go beyond the reach of prior PSRO applications by considering 3- to 5-player poker games, yielding instances where α-Rank achieves faster convergence than approximate Nash solvers, thus establishing it as a favorable general games solver. We also carry out an initial empirical validation in MuJoCo soccer, illustrating the feasibility of the proposed approach in another complex domain.
| Type: | Proceedings paper |
|---|---|
| Title: | A Generalized Training Approach for Multiagent Learning |
| Event: | 8th International Conference on Learning Representations, ICLR 2020 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://openreview.net/group?id=ICLR.cc/2020/Confe... |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher's terms and conditions. |
| Keywords: | multiagent learning, game theory, training, games |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10109593 |
Archive Staff Only
 |
View Item |
