eprintid: 1561645 rev_number: 46 eprint_status: archive userid: 608 dir: disk0/01/56/16/45 datestamp: 2017-07-18 11:07:18 lastmod: 2021-09-20 00:05:13 status_changed: 2017-10-13 11:29:13 type: article metadata_visibility: show creators_name: Sokolic, J creators_name: Giryes, R creators_name: Sapiro, G creators_name: Rodrigues, MRD title: Robust Large Margin Deep Neural Networks ispublished: pub divisions: UCL divisions: B04 divisions: C05 divisions: F46 keywords: Science & Technology, Technology, Engineering, Electrical & Electronic, Engineering, Deep Learning, Deep Neural Networks, Generalization Error, Robustness, Sensitivity-Analysis, Recognition note: This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/ abstract: The generalization error of deep neural networks via their classification margin is studied in this paper. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary nonlinearities and pooling layers, and to networks with different architectures such as feed forward networks and residual networks. Our analysis leads to the conclusion that a bounded spectral norm of the network's Jacobian matrix in the neighbourhood of the training samples is crucial for a deep neural network of arbitrary depth and width to generalize well. This is a significant improvement over the current bounds in the literature, which imply that the generalization error grows with either the width or the depth of the network. Moreover, it shows that the recently proposed batch normalization and weight normalization reparametrizations enjoy good generalization properties, and leads to a novel network regularizer based on the network's Jacobian matrix. The analysis is supported with experimental results on the MNIST, CIFAR-10, LaRED, and ImageNet datasets. date: 2017-08-15 date_type: published publisher: IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC official_url: http://dx.doi.org/10.1109/TSP.2017.2708039 oa_status: green full_text_type: pub language: eng primo: open primo_central: open_green article_type_text: Article verified: verified_manual elements_id: 1301079 doi: 10.1109/TSP.2017.2708039 lyricists_name: Rodrigues, Miguel lyricists_name: Sokolic, Jure lyricists_id: MRDIA06 lyricists_id: JSOKO82 actors_name: Rodrigues, Miguel actors_id: MRDIA06 actors_role: owner full_text_status: public publication: IEEE Transactions on Signal Processing volume: 65 number: 16 pagerange: 4265-4280 pages: 16 issn: 1941-0476 citation: Sokolic, J; Giryes, R; Sapiro, G; Rodrigues, MRD; (2017) Robust Large Margin Deep Neural Networks. IEEE Transactions on Signal Processing , 65 (16) pp. 4265-4280. 10.1109/TSP.2017.2708039 <https://doi.org/10.1109/TSP.2017.2708039>. Green open access document_url: https://discovery.ucl.ac.uk/id/eprint/1561645/1/Rodrigues_07934087.pdf