Clerico, Eugenio;
Guedj, Benjamin;
(2024)
A note on regularised NTK dynamics with an application to PAC-Bayesian training.
Transactions on Machine Learning Research
, 2024
(04)
pp. 1-20.
Preview |
Text
1972_A_note_on_regularised_NTK.pdf - Published Version Download (470kB) | Preview |
Abstract
We establish explicit dynamics for neural networks whose training objective has a regularising term that constrains the parameters to remain close to their initial value. This keeps the network in a lazy training regime, where the dynamics can be linearised around the initialisation. The standard neural tangent kernel (NTK) governs the evolution during the training in the infinite-width limit, although the regularisation yields an additional term that appears in the differential equation describing the dynamics. This setting provides an appropriate framework to study the evolution of wide networks trained to optimise generalisation objectives such as PAC-Bayes bounds, and hence contribute to a deeper theoretical understanding of such networks.
| Type: | Article |
|---|---|
| Title: | A note on regularised NTK dynamics with an application to PAC-Bayesian training |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://openreview.net/forum?id=2la55BeWwy |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher's terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10196071 |
Archive Staff Only
 |
View Item |
