Sargent, Matthew James; (2025) Structure as an Inductive Bias for Reinforcement Learning. Doctoral thesis (Ph.D), UCL (University College London).

Text Matthew_James_Sargent_Corrected_Thesis.pdf - Accepted Version Access restricted to UCL open access staff until 1 April 2026. Download (33MB)

Abstract

Reinforcement learning (RL) has achieved remarkable successes in recent years, yet agents often require extensive interaction with their environment to master complex tasks, and their learned policies frequently fail to generalise beyond the training distribution. This thesis demonstrates how \emph{structural inductive biases} can mitigate these limitations by enabling more efficient exploration and facilitating robust policy transfer. We investigate several methods for injecting structure into RL, beginning with temporally abstracted random walks for exploration, where sampling action repeats from a heavy-tailed distribution grants improved coverage of the state space over conventional $\epsilon$-greedy strategies. We quantify the how structure contributes to the success of these strategies through graph curvature. Next, we introduce \emph{temporally extended Successor Representations}, which incorporate action skips into the SR framework, thereby enhancing credit assignment over longer horizons and improving sample efficiency. We then exploit state-space symmetries to compress successor representations via graph automorphisms, reducing memory requirements without degrading performance. In addition, we propose a meta-learning approach for rapidly inferring the Laplacian eigenvectors of novel environments, thus capturing crucial structural properties of the transition graph and accelerating adaptation to new tasks. Our empirical evaluations span discrete grid worlds, MiniGrid-based environments, and transformer-based RL models, consistently showing that these structural priors foster faster learning and better generalisation. An analysis of emergent representations in transformer-based RL further reveals how these models implicitly encode and utilise environmental structure. Collectively, this thesis underscores the potential of incorporating structural inductive biases into RL and presents novel techniques for leveraging the underlying geometry and symmetries of the state space to enhance agent performance and efficiency.

Download activity - last month

Export as XMLJSONCSV

Download activity - last 12 months

Export as XMLJSONCSV

Downloads by country - last 12 months

Export as JSONXMLCSV

Archive Staff Only

View Item