UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Successor-Predecessor Intrinsic Exploration

Yu, Changmin; Burgess, neil; Sahani, Maneesh; Gershman, Samuel J; (2023) Successor-Predecessor Intrinsic Exploration. OpenReview.net: Amherst, MA, United States. Green open access

[thumbnail of Sahani_Successor-Predecessor Intrinsic Exploration_pre-print.pdf]
Sahani_Successor-Predecessor Intrinsic Exploration_pre-print.pdf

Download (4MB) | Preview


Exploration is essential in reinforcement learning, particularly in environments where external rewards are sparse. Here we focus on exploration with intrinsic rewards, where the agent transiently augments the external rewards with self-generated intrinsic rewards. Although the study of intrinsic rewards has a long history, existing methods focus on composing the intrinsic reward based on measures of future prospects of states, ignoring the information contained in the retrospective structure of transition sequences. Here we argue that the agent can utilise retrospective information to generate explorative behaviour with structure-awareness, facilitating efficient exploration based on global instead of local information. We propose Successor-Predecessor Intrinsic Exploration (SPIE), an exploration algorithm based on a novel intrinsic reward combining prospective and retrospective information. We show that SPIE yields more efficient and ethologically plausible exploratory behaviour in environments with sparse rewards and bottleneck states than competing methods. We also implement SPIE in deep reinforcement learning agents, and show that the resulting agent achieves stronger empirical performance than existing methods on sparse-reward Atari games.

Type: Working / discussion paper
Title: Successor-Predecessor Intrinsic Exploration
Open access status: An open access version is available from UCL Discovery
Publisher version: https://openreview.net/forum?id=ohKbQp0jIY
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher's terms and conditions.
Keywords: Exploration, reinforcement learning
UCL classification: UCL
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Gatsby Computational Neurosci Unit
URI: https://discovery.ucl.ac.uk/id/eprint/10181229
Downloads since deposit
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item