Wen, Y;
Chen, H;
Yang, Y;
Li, M;
Tian, Z;
Chen, X;
Wang, J;
(2023)
A Game-Theoretic Approach to Multi-agent Trust Region Optimization.
In: Yokoo, M and Qiao, H and Vorobeychik, Y and Hao, J, (eds.)
International Conference on Distributed Artificial Intelligence DAI 2022: Distributed Artificial Intelligence.
(pp. pp. 74-87).
Springer, Cham
Preview |
Text
2106.06828-2.pdf - Other Download (3MB) | Preview |
Abstract
Trust region methods are widely applied in single-agent reinforcement learning problems due to their monotonic performance-improvement guarantee at every iteration. Nonetheless, when applied in multi-agent settings, the guarantee of trust region methods no longer holds because an agent’s payoff is also affected by other agents’ adaptive behaviors. To tackle this problem, we conduct a game-theoretical analysis in the policy space, and propose a multi-agent trust region learning method (MATRL), which enables trust region optimization for multi-agent learning. Specifically, MATRL finds a stable improvement direction that is guided by the solution concept of Nash equilibrium at the meta-game level. We derive the monotonic improvement guarantee in multi-agent settings and show the local convergence of MATRL to stable fixed points in differential games. To test our method, we evaluate MATRL in both discrete and continuous multiplayer general-sum games including checker and switch grid worlds, multi-agent MuJoCo, and Atari games. Results suggest that MATRL significantly outperforms strong multi-agent reinforcement learning baselines.
| Type: | Proceedings paper |
|---|---|
| Title: | A Game-Theoretic Approach to Multi-agent Trust Region Optimization |
| Event: | Distributed Artificial Intelligence 4th International Conference, DAI 2022 |
| Location: | Tianjin, PEOPLES R CHINA |
| Dates: | 15 Dec 2022 - 17 Dec 2022 |
| ISBN-13: | 9783031255489 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1007/978-3-031-25549-6_6 |
| Publisher version: | https://doi.org/10.1007/978-3-031-25549-6_6 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Science & Technology, Technology, Computer Science, Artificial Intelligence, Computer Science, Interdisciplinary Applications, Computer Science, Multi-agent Reinforcement Learning, Game Theory, Trust Region Optimization |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10173720 |
Archive Staff Only
 |
View Item |
