UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

TreeqN and ATreEC: Differentiable tree-structured models for deep reinforcement learning

Farquhar, G; Rocktäschel, T; Igl, M; Whiteson, S; (2018) TreeqN and ATreEC: Differentiable tree-structured models for deep reinforcement learning. In: Proceedings of 6th International Conference on Learning Representations (ICLR 2018). ICLR: Vancouver, BC, Canada. Green open access

[thumbnail of 1710.11417v2.pdf]
Preview
Text
1710.11417v2.pdf - Published Version

Download (4MB) | Preview

Abstract

© Learning Representations, ICLR 2018 - Conference Track Proceedings.All right reserved. Combining deep model-free reinforcement learning with on-line planning is a promising approach to building on the successes of deep RL. On-line planning with look-ahead trees has proven successful in environments where transition models are known a priori. However, in complex environments where transition models need to be learned from data, the deficiencies of learned models have limited their utility for planning. To address these challenges, we propose TreeQN, a differentiable, recursive, tree-structured model that serves as a drop-in replacement for any value function network in deep RL with discrete actions. TreeQN dynamically constructs a tree by recursively applying a transition model in a learned abstract state space and then aggregating predicted rewards and state-values using a tree backup to estimate Q-values. We also propose ATreeC, an actor-critic variant that augments TreeQN with a softmax layer to form a stochastic policy network. Both approaches are trained end-to-end, such that the learned model is optimised for its actual use in the tree. We show that TreeQN and ATreeC outperform n-step DQN and A2C on a box-pushing task, as well as n-step DQN and value prediction networks (Oh et al., 2017) on multiple Atari games. Furthermore, we present ablation studies that demonstrate the effect of different auxiliary losses on learning transition models.

Type: Proceedings paper
Title: TreeqN and ATreEC: Differentiable tree-structured models for deep reinforcement learning
Event: 6th International Conference on Learning Representations (ICLR 2018)
Open access status: An open access version is available from UCL Discovery
Publisher version: https://iclr.cc/archive/www/doku.php%3Fid=iclr2018...
Language: English
Additional information: This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10074783
Downloads since deposit
18Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item