UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Lessons from the Rademacher complexity for deep learning

Sokolic, J; Gyries, R; Sapiro, G; Rodrigues, MRD; (2016) Lessons from the Rademacher complexity for deep learning. In: Bengio, Samy and Kingsbury, Brian, (eds.) Proceedings of the workshop track - International Conference on Learning Representations 2016: ICLR 2016. ICLR: San Juan, Puerto Rico. Green open access

[thumbnail of LessonsFromTheRCforDL_Sokolic.pdf]
Preview
Text
LessonsFromTheRCforDL_Sokolic.pdf - Published Version

Download (264kB) | Preview

Abstract

Understanding the generalization properties of deep learning models is critical for successful applications, especially in the regimes where the number of training samples is limited. We study the generalization properties of deep neural networks via the empirical Rademacher complexity and show that it is easier to control the complexity of convolutional networks compared to general fully connected networks. In particular, we justify the usage of small convolutional kernels in deep networks as they lead to a better generalization error. Moreover, we propose a representation based regularization method that allows to decrease the generalization error by controlling the coherence of the representation. Experiments on the MNIST dataset support these foundations.

Type: Proceedings paper
Title: Lessons from the Rademacher complexity for deep learning
Event: 2016 International Conference on Learning Representations (ICLR)
Open access status: An open access version is available from UCL Discovery
Publisher version: https://iclr.cc/archive/www/doku.php%3Fid=iclr2016...
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Electronic and Electrical Eng
URI: https://discovery.ucl.ac.uk/id/eprint/10063246
Downloads since deposit
117Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item