Mavor-Parker, Augustine N;
Young, Kimberly A;
Barry, Caswell;
Griffin, Lewis D;
(2022)
How to Stay Curious while avoiding Noisy TVs using Aleatoric Uncertainty Estimation.
In: Chaudhuri, K and Jegelka, S and Song, L and Szepesvari, C and Niu, G and Sabato, S, (eds.)
International Conference on Machine Learning.
PMLR
Preview |
PDF
mavor-parker22a.pdf - Published Version Download (6MB) | Preview |
Abstract
When extrinsic rewards are sparse, artificial agents struggle to explore an environment. Curiosity, implemented as an intrinsic reward for prediction errors, can improve exploration but it is known to fail when faced with action-dependent noise sources (‘noisy TVs’). In an attempt to make exploring agents robust to noisy TVs, we present a simple solution: aleatoric mapping agents (AMAs). AMAs are a novel form of curiosity that explicitly ascertain which state transitions of the environment are unpredictable, even if those dynamics are induced by the actions of the agent. This is achieved by generating separate forward predictions for the mean and aleatoric uncertainty of future states, with the aim of reducing intrinsic rewards for those transitions that are unpredictable. We demonstrate that in a range of environments AMAs are able to circumvent actiondependent stochastic traps that immobilise conventional curiosity driven agents. Furthermore, we demonstrate empirically that other common exploration approaches—previously thought to be immune to agent-induced randomness—can be trapped by stochastic dynamics. Code to reproduce our experiments is provided.
| Type: | Proceedings paper |
|---|---|
| Title: | How to Stay Curious while avoiding Noisy TVs using Aleatoric Uncertainty Estimation |
| Event: | 38th International Conference on Machine Learning (ICML) |
| Location: | Baltimore, MD |
| Dates: | 17 Jul 2022 - 23 Jul 2022 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://proceedings.mlr.press/v162/ |
| Language: | English |
| Additional information: | This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third-party material in this article are included in the Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ |
| Keywords: | Science & Technology, Technology, Computer Science, Artificial Intelligence, Computer Science, ARCADE LEARNING-ENVIRONMENT, ACETYLCHOLINE, RELEASE, HABITUATION, NOVELTY |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10170013 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
