Letcher, A;
Foerster, J;
Balduzzi, D;
Rocktäschel, T;
Whiteson, S;
(2019)
Stable Opponent Shaping in Differentiable Games.
In:
Proceedings of the 7th International Conference on Learning Representations:ICLR 2019.
ICLR: New Orleans, Louisiana,USA.
Preview |
Text
1811.08469v2.pdf - Accepted Version Download (3MB) | Preview |
Abstract
A growing number of learning methods are actually differentiable games whose players optimise multiple, interdependent objectives in parallel -- from GANs and intrinsic curiosity to multi-agent RL. Opponent shaping is a powerful approach to improve learning dynamics in these games, accounting for player influence on others' updates. Learning with Opponent-Learning Awareness (LOLA) is a recent algorithm that exploits this response and leads to cooperation in settings like the Iterated Prisoner's Dilemma. Although experimentally successful, we show that LOLA agents can exhibit 'arrogant' behaviour directly at odds with convergence. In fact, remarkably few algorithms have theoretical guarantees applying across all (n-player, non-convex) games. In this paper we present Stable Opponent Shaping (SOS), a new method that interpolates between LOLA and a stable variant named LookAhead. We prove that LookAhead converges locally to equilibria and avoids strict saddles in all differentiable games. SOS inherits these essential guarantees, while also shaping the learning of opponents and consistently either matching or outperforming LOLA experimentally.
| Type: | Proceedings paper |
|---|---|
| Title: | Stable Opponent Shaping in Differentiable Games |
| Event: | 7th International Conference on Learning Representations:ICLR 2019 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://iclr.cc/Conferences/2019/Schedule?type=Pos... |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | multi-agent learning, multiple interacting losses, opponent shaping, exploitation, convergence |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10074412 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
