Sootla, A;
Cowen-Rivers, AI;
Wang, J;
Ammar, HB;
(2022)
Enhancing Safe Exploration Using Safety State Augmentation.
In:
Advances in Neural Information Processing Systems.
Preview |
Text
6142_enhancing_safe_exploration_usi.pdf - Published Version Download (2MB) | Preview |
Abstract
Safe exploration is a challenging and important problem in model-free reinforcement learning (RL). Often the safety cost is sparse and unknown, which unavoidably leads to constraint violations - a phenomenon ideally to be avoided in safety-critical applications. We tackle this problem by augmenting the state-space with a safety state, which is nonnegative if and only if the constraint is satisfied. The value of this state also serves as a distance toward constraint violation, while its initial value indicates the available safety budget. This idea allows us to derive policies for scheduling the safety budget during training. We call our approach Simmer (Safe policy IMproveMEnt for RL) to reflect the careful nature of these schedules. We apply this idea to two safe RL problems: RL with constraints imposed on an average cost, and RL with constraints imposed on a cost with probability one. Our experiments suggest that “simmering” a safe algorithm can improve safety during training for both settings. We further show that Simmer can stabilize training and improve the performance of safe RL with average constraints.
| Type: | Proceedings paper |
|---|---|
| Title: | Enhancing Safe Exploration Using Safety State Augmentation |
| Event: | 36th Conference on Neural Information Processing Systems (NeurIPS 2022) |
| ISBN-13: | 9781713871088 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://proceedings.neurips.cc/paper_files/paper/2... |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10173241 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
