eprintid: 10196071
rev_number: 8
eprint_status: archive
userid: 699
dir: disk0/10/19/60/71
datestamp: 2024-08-22 10:02:11
lastmod: 2024-08-22 10:02:11
status_changed: 2024-08-22 10:02:11
type: article
metadata_visibility: show
sword_depositor: 699
creators_name: Clerico, Eugenio
creators_name: Guedj, Benjamin
title: A note on regularised NTK dynamics with an application to PAC-Bayesian training
ispublished: pub
divisions: UCL
divisions: B04
divisions: F48
note: This version is the version of record. For information on re-use, please refer to the publisher's terms and conditions.
abstract: We establish explicit dynamics for neural networks whose training objective has a regularising term that constrains the parameters to remain close to their initial value. This keeps the network in a lazy training regime, where the dynamics can be linearised around the initialisation. The standard neural tangent kernel (NTK) governs the evolution during the training in the infinite-width limit, although the regularisation yields an additional term that appears in the differential equation describing the dynamics. This setting provides an appropriate framework to study the evolution of wide networks trained to optimise generalisation objectives such as PAC-Bayes bounds, and hence contribute to a deeper theoretical understanding of such networks.
date: 2024
date_type: published
official_url: https://openreview.net/forum?id=2la55BeWwy
oa_status: green
full_text_type: pub
language: eng
primo: open
primo_central: open_green
verified: verified_manual
elements_id: 2306430
lyricists_name: Guedj, Benjamin
lyricists_id: BGUED94
actors_name: Flynn, Bernadette
actors_id: BFFLY94
actors_role: owner
full_text_status: public
publication: Transactions on Machine Learning Research
volume: 2024
number: 04
pagerange: 1-20
citation:        Clerico, Eugenio;    Guedj, Benjamin;      (2024)    A note on regularised NTK dynamics with an application to PAC-Bayesian training.                   Transactions on Machine Learning Research , 2024  (04)   pp. 1-20.          Green open access   
 
document_url: https://discovery.ucl.ac.uk/id/eprint/10196071/1/1972_A_note_on_regularised_NTK.pdf