Acero, F;
Li, Z;
(2024)
Distilling Reinforcement Learning Policies for Interpretable Robot Locomotion: Gradient Boosting Machines and Symbolic Regression.
In:
IEEE International Conference on Intelligent Robots and Systems.
(pp. pp. 6840-6847).
IEEE
Preview |
Text
Distilling_Reinforcement_Learning_Policies_for_Interpretable_Robot_Locomotion__Gradient_Boosting_Machines_and_Symbolic_Regression.pdf - Accepted Version Download (9MB) | Preview |
Abstract
Recent advancements in reinforcement learning (RL) have led to remarkable achievements in robot locomotion capabilities. However, the complexity and "black-box"nature of neural network-based RL policies hinder their interpretability and broader acceptance, particularly in applications demanding high levels of safety and reliability. This paper introduces a novel approach to distill neural RL policies into more interpretable forms using Gradient Boosting Machines (GBMs), Explainable Boosting Machines (EBMs) and Symbolic Regression. By leveraging the inherent interpretability of generalized additive models, decision trees, and analytical expressions, we transform opaque neural network policies into more transparent "glass-box"models. We train expert neural network policies using RL and subsequently distill them into (i) GBMs, (ii) EBMs, and (iii) symbolic policies. To address the inherent distribution shift challenge of behavioral cloning, we propose to use the Dataset Aggregation (DAgger) algorithm with a curriculum of episode-dependent alternation of actions between expert and distilled policies, to enable efficient distillation of feedback control policies. We evaluate our approach on various robot locomotion gaits - walking, trotting, bounding, and pacing - and study the importance of different observations in joint actions for distilled policies using various methods. We train neural expert policies for 205 hours of simulated experience and distill interpretable policies with only 10 minutes of simulated interaction for each gait using the proposed method.
| Type: | Proceedings paper |
|---|---|
| Title: | Distilling Reinforcement Learning Policies for Interpretable Robot Locomotion: Gradient Boosting Machines and Symbolic Regression |
| Event: | 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) |
| Location: | Abu Dhabi, United Arab Emirates |
| Dates: | 14th-18th October 2024 |
| ISBN-13: | 979-8-3503-7771-2 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1109/IROS58592.2024.10802433 |
| Publisher version: | https://doi.org/10.1109/iros58592.2024.10802433 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher's terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10210063 |
Archive Staff Only
 |
View Item |
