Dominé, Clémentine Carla Juliette; (2025) Balancing Learning Regimes: The Impact of Prior Knowledge on the Dynamics of Neural Representations. Doctoral thesis (Ph.D), UCL (University College London).

Preview

Text Clementine_Carla_Juliette_Domine_20032697_PhD_Thesis.pdf - Accepted Version Download (7MB) | Preview

Abstract

Biological and artificial neural networks create internal representations that support complex tasks like reasoning and decision-making. While characterising these representations has been a key focus in machine learning and neuroscience, the mechanisms for extracting task-specific features and the influence of prior knowledge remain unclear. To gain insight, it’s essential to analyse the interactions between datasets, network architectures and initialisation. Previous work has shown that specific initialisations place networks in a ‘lazy’ regime, where internal representations do not change through learning, while other initialisations place networks in a ‘rich/feature learning’ regime, where internal representations evolve to fit a task. Here, we study how initialisation and architecture learning structured data influence learning dynamics in deep neural networks. First, we derive novel exact solutions for deep linear networks with ‘lambda-balanced’ initialisations that differ in the norms of the weights across layers, which approximate common initialisations used in practice. Our results show that imbalanced initialisations lead to a lazy learning regime, while balanced ones promote a rich regime. These findings enhance the understanding of how weight initialisation and network structure influence learning, with implications for continual, reversal, and transfer learning in neuroscience and practical applications. Next, we demonstrate that our theoretical findings, derived from deep linear networks, have significant implications for non-linear networks. Utilising the non-linear teacher-student theoretical framework for neural network analysis, we reveal a strong dependence of specialisation—characterised by rich, task-specific representations—on initial weight imbalance. We discuss the implications of this understanding in the context of continual learning and showcase its application in practical machine learning scenarios, such as grokking, developing edge detectors in convolutional neural networks, and neuroscience. Overall, our results highlight the critical role of initialisation imbalance in the learning dynamics of both artificial and biological neural networks.

Download activity - last month

Export as JSONCSVXML

Download activity - last 12 months

Export as XMLCSVJSON

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item