eprintid: 10198782
rev_number: 10
eprint_status: archive
userid: 699
dir: disk0/10/19/87/82
datestamp: 2024-10-23 14:17:50
lastmod: 2024-10-23 14:17:50
status_changed: 2024-10-23 14:17:50
type: proceedings_section
metadata_visibility: show
sword_depositor: 699
creators_name: Ramesh, Shyam Sundhar
creators_name: Sessa, Pier Giuseppe
creators_name: Hu, Yifan
creators_name: Krause, Andreas
creators_name: Bogunovic, Ilija
title: Distributionally Robust Model-based Reinforcement Learning with Large State Spaces
ispublished: pub
divisions: UCL
divisions: B04
divisions: F46
note: © The Author 2024. Original content in this paper is licensed under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0) Licence (https://creativecommons.org/licenses/by/4.0/).
abstract: Three major challenges in reinforcement learning are the complex dynamical systems with large state spaces, the costly data acquisition processes, and the deviation of real-world dynamics from the training environment deployment. To overcome these issues, we study distributionally robust Markov decision processes with continuous state spaces under the widely used Kullback–Leibler, chi-square, and total variation uncertainty sets. We propose a model-based approach that utilizes Gaussian Processes and the maximum variance reduction algorithm to efficiently learn multi-output nominal transition dynamics, leveraging access to a generative model (i.e., simulator). We further demonstrate the statistical sample complexity of the proposed method for different uncertainty sets. These complexity bounds are independent of the number of states and extend beyond linear dynamics, ensuring the effectiveness of our approach in identifying near-optimal distributionally-robust policies. The proposed method can be further combined with other model-free distributionally robust reinforcement learning methods to obtain a near-optimal robust policy. Experimental results demonstrate the robustness of our algorithm to distributional shifts and its superior performance in terms of the number of samples needed.
date: 2024
date_type: published
publisher: PMLR
official_url: https://proceedings.mlr.press/v238/sundhar-ramesh24a.html
oa_status: green
full_text_type: pub
language: eng
primo: open
primo_central: open_green
verified: verified_manual
elements_id: 2282805
lyricists_name: Bogunovic, Ilija
lyricists_name: Ramesh, Shyam Sundhar
lyricists_id: IBOGU49
lyricists_id: SRAME02
actors_name: Ramesh, Shyam Sundhar
actors_id: SRAME02
actors_role: owner
full_text_status: public
pres_type: paper
publication: INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE AND STATISTICS, VOL 238
volume: 238
pagerange: 1-42
pages: 42
event_title: 27th International Conference on Artificial Intelligence and Statistics (AISTATS)
event_location: Valencia, Spain
event_dates: 2nd-4th May 2024
book_title: Proceedings of The 27th International Conference on Artificial Intelligence and Statistics
editors_name: Dasgupta, S
editors_name: Mandt, S
editors_name: Li, Y
citation:        Ramesh, Shyam Sundhar;    Sessa, Pier Giuseppe;    Hu, Yifan;    Krause, Andreas;    Bogunovic, Ilija;      (2024)    Distributionally Robust Model-based Reinforcement Learning with Large State Spaces.                     In: Dasgupta, S and Mandt, S and Li, Y, (eds.) Proceedings of The 27th International Conference on Artificial Intelligence and Statistics.  (pp. pp. 1-42).  PMLR       Green open access   
 
document_url: https://discovery.ucl.ac.uk/id/eprint/10198782/1/sundhar-ramesh24a.pdf