Braun, Lukas;
Dominé, Clémentine Carla Juliette;
Fitzgerald, James E;
Saxe, Andrew M;
(2022)
Exact learning dynamics of deep linear networks with prior knowledge.
In:
Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
NeurIPS
Preview |
Text
exact_learning_dynamics_of_dee.pdf - Published Version Download (3MB) | Preview |
Abstract
Learning in deep neural networks is known to depend critically on the knowledge embedded in the initial network weights. However, few theoretical results have precisely linked prior knowledge to learning dynamics. Here we derive exact solutions to the dynamics of learning with rich prior knowledge in deep linear networks by generalising Fukumizu’s matrix Riccati solution \textbackslashcitepfukumizu1998effect. We obtain explicit expressions for the evolving network function, hidden representational similarity, and neural tangent kernel over training for a broad class of initialisations and tasks. The expressions reveal a class of task-independent initialisations that radically alter learning dynamics from slow non-linear dynamics to fast exponential trajectories while converging to a global optimum with identical representational similarity, dissociating learning trajectories from the structure of initial internal representations. We characterise how network weights dynamically align with task structure, rigorously justifying why previous solutions successfully described learning from small initial weights without incorporating their fine-scale structure. Finally, we discuss the implications of these findings for continual learning, reversal learning and learning of structured knowledge. Taken together, our results provide a mathematical toolkit for understanding the impact of prior knowledge on deep learning.
Type: | Proceedings paper |
---|---|
Title: | Exact learning dynamics of deep linear networks with prior knowledge |
Event: | 36th Conference on Neural Information Processing Systems (NeurIPS 2022) |
Open access status: | An open access version is available from UCL Discovery |
Publisher version: | https://openreview.net/forum?id=lJx2vng-KiC |
Language: | English |
Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Gatsby Computational Neurosci Unit |
URI: | https://discovery.ucl.ac.uk/id/eprint/10160780 |
Archive Staff Only
![]() |
View Item |