Mavor-Parker, Augustine N.; (2025) Investigating Sample Efficient Deep Reinforcement Learning. Doctoral thesis (Ph.D), UCL (University College London).

Preview

Text Mavor-Parker_Thesis.pdf - Accepted Version Download (13MB) | Preview

Abstract

Deep reinforcement learning (RL) has achieved superhuman performance on games [Silver et al., 2017], learned to control nuclear fusion reactors [Degrave et al., 2022] and can be used to adapt foundation models to human preferences [Achiam et al., 2023]. However, adoption in the real world is hindered by the interconnected problems of sample efficiency and generalisation. In this thesis we study three methods for improving the sample efficiency of deep RL agents: curiosity driven exploration, abstractions of environment states-action pairs and network architectures that learn value functions more quickly. We improve the robustness of curiosity driven agents [Pathak et al., 2017] to noisy transitions by enabling agents to predict the amount of randomness in their environment and then decrease their curiosity about transitions that are predicted to be stochastic. Next, to reduce the number of states an RL agent is required to explore, we learn forwards and backwards models of the environment to abstract away equivalent state-action pairs, which we formalise with MDP homomorphisms [Ravindran, 2004]. Finally, we examine the use of periodic activation functions to improve sample efficiency [Li and Pathak, 2021, Yang et al., 2022], showing that representation frequencies become exceedingly large and hence generalise poorly. To improve their generalisation, we suggest penalising the magnitude of the frequency of periodic representations during learning.

Download activity - last month

Export as JSONCSVXML

Download activity - last 12 months

Export as CSVXMLJSON

Downloads by country - last 12 months

Export as JSONXMLCSV

Archive Staff Only

View Item