UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Robust Large Margin Deep Neural Networks

Sokolic, J; Giryes, R; Sapiro, G; Rodrigues, MRD; (2017) Robust Large Margin Deep Neural Networks. IEEE Transactions on Signal Processing , 65 (16) pp. 4265-4280. 10.1109/TSP.2017.2708039. Green open access

[thumbnail of Rodrigues_07934087.pdf]
Rodrigues_07934087.pdf - Published Version

Download (1MB) | Preview


The generalization error of deep neural networks via their classification margin is studied in this paper. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary nonlinearities and pooling layers, and to networks with different architectures such as feed forward networks and residual networks. Our analysis leads to the conclusion that a bounded spectral norm of the network's Jacobian matrix in the neighbourhood of the training samples is crucial for a deep neural network of arbitrary depth and width to generalize well. This is a significant improvement over the current bounds in the literature, which imply that the generalization error grows with either the width or the depth of the network. Moreover, it shows that the recently proposed batch normalization and weight normalization reparametrizations enjoy good generalization properties, and leads to a novel network regularizer based on the network's Jacobian matrix. The analysis is supported with experimental results on the MNIST, CIFAR-10, LaRED, and ImageNet datasets.

Type: Article
Title: Robust Large Margin Deep Neural Networks
Open access status: An open access version is available from UCL Discovery
DOI: 10.1109/TSP.2017.2708039
Publisher version: http://dx.doi.org/10.1109/TSP.2017.2708039
Language: English
Additional information: This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/
Keywords: Science & Technology, Technology, Engineering, Electrical & Electronic, Engineering, Deep Learning, Deep Neural Networks, Generalization Error, Robustness, Sensitivity-Analysis, Recognition
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Electronic and Electrical Eng
URI: https://discovery.ucl.ac.uk/id/eprint/1561645
Downloads since deposit
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item