Gouk, Henry;
Hospedales, Timothy M;
Pontil, Massimiliano;
(2021)
Distance-Based Regularisation of Deep Networks for Fine-Tuning.
In:
Proceedings of the International Conference on Learning Representations ICLR 2021.
ICLR
Preview |
Text
2216_distance_based_regularisation_.pdf - Published Version Download (269kB) | Preview |
Abstract
We investigate approaches to regularisation during fine-tuning of deep neural networks. First we provide a neural network generalisation bound based on Rademacher complexity that uses the distance the weights have moved from their initial values. This bound has no direct dependence on the number of weights and compares favourably to other bounds when applied to convolutional networks. Our bound is highly relevant for fine-tuning, because providing a network with a good initialisation based on transfer learning means that learning can modify the weights less, and hence achieve tighter generalisation. Inspired by this, we develop a simple yet effective fine-tuning algorithm that constrains the hypothesis class to a small sphere centred on the initial pre-trained weights, thus obtaining provably better generalisation performance than conventional transfer learning. Empirical evaluation shows that our algorithm works well, corroborating our theoretical results. It outperforms both state of the art fine-tuning competitors, and penalty-based alternatives that we show do not directly constrain the radius of the search space.
| Type: | Proceedings paper |
|---|---|
| Title: | Distance-Based Regularisation of Deep Networks for Fine-Tuning |
| Event: | the International Conference on Learning Representations ICLR 2021 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://openreview.net/group?id=ICLR.cc/2021/Confe... |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10164239 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
