TY  - GEN
N1  - This version is the version of record. For information on re-use, please refer to the publisher?s terms and conditions.
AV  - public
Y1  - 2021///
TI  - Distance-Based Regularisation of Deep Networks for Fine-Tuning
A1  - Gouk, Henry
A1  - Hospedales, Timothy M
A1  - Pontil, Massimiliano
PB  - ICLR
UR  - https://openreview.net/group?id=ICLR.cc/2021/Conference
N2  - We investigate approaches to regularisation during fine-tuning of deep neural
networks. First we provide a neural network generalisation bound based on
Rademacher complexity that uses the distance the weights have moved from their
initial values. This bound has no direct dependence on the number of weights and
compares favourably to other bounds when applied to convolutional networks. Our
bound is highly relevant for fine-tuning, because providing a network with a good
initialisation based on transfer learning means that learning can modify the weights
less, and hence achieve tighter generalisation. Inspired by this, we develop a simple
yet effective fine-tuning algorithm that constrains the hypothesis class to a small
sphere centred on the initial pre-trained weights, thus obtaining provably better
generalisation performance than conventional transfer learning. Empirical evaluation shows that our algorithm works well, corroborating our theoretical results.
It outperforms both state of the art fine-tuning competitors, and penalty-based
alternatives that we show do not directly constrain the radius of the search space.
ID  - discovery10164239
ER  -