eprintid: 10164239 rev_number: 7 eprint_status: archive userid: 699 dir: disk0/10/16/42/39 datestamp: 2023-02-03 13:52:09 lastmod: 2023-02-03 13:52:09 status_changed: 2023-02-03 13:52:09 type: proceedings_section metadata_visibility: show sword_depositor: 699 creators_name: Gouk, Henry creators_name: Hospedales, Timothy M creators_name: Pontil, Massimiliano title: Distance-Based Regularisation of Deep Networks for Fine-Tuning ispublished: pub divisions: UCL divisions: B04 divisions: C05 divisions: F48 note: This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. abstract: We investigate approaches to regularisation during fine-tuning of deep neural networks. First we provide a neural network generalisation bound based on Rademacher complexity that uses the distance the weights have moved from their initial values. This bound has no direct dependence on the number of weights and compares favourably to other bounds when applied to convolutional networks. Our bound is highly relevant for fine-tuning, because providing a network with a good initialisation based on transfer learning means that learning can modify the weights less, and hence achieve tighter generalisation. Inspired by this, we develop a simple yet effective fine-tuning algorithm that constrains the hypothesis class to a small sphere centred on the initial pre-trained weights, thus obtaining provably better generalisation performance than conventional transfer learning. Empirical evaluation shows that our algorithm works well, corroborating our theoretical results. It outperforms both state of the art fine-tuning competitors, and penalty-based alternatives that we show do not directly constrain the radius of the search space. date: 2021 date_type: published publisher: ICLR official_url: https://openreview.net/group?id=ICLR.cc/2021/Conference oa_status: green full_text_type: pub language: eng primo: open primo_central: open_green verified: verified_manual elements_id: 1873397 lyricists_name: Pontil, Massimiliano lyricists_id: MPONT27 actors_name: Pontil, Massimiliano actors_id: MPONT27 actors_role: owner full_text_status: public pres_type: paper publication: ICLR event_title: the International Conference on Learning Representations ICLR 2021 book_title: Proceedings of the International Conference on Learning Representations ICLR 2021 citation: Gouk, Henry; Hospedales, Timothy M; Pontil, Massimiliano; (2021) Distance-Based Regularisation of Deep Networks for Fine-Tuning. In: Proceedings of the International Conference on Learning Representations ICLR 2021. ICLR Green open access document_url: https://discovery.ucl.ac.uk/id/eprint/10164239/1/2216_distance_based_regularisation_.pdf