Frecon, Jordan;
Gasso, Gilles;
Pontil, Massimiliano;
Salzo, Saverio;
(2022)
Bregman Neural Networks.
In:
Proceedings of Machine Learning Research.
(pp. pp. 6779-6792).
Proceedings of Machine Learning Research (PMLR)
Preview |
PDF
frecon22a.pdf - Published Version Download (757kB) | Preview |
Abstract
We present a framework based on bilevel optimization for learning multilayer, deep data representations. On the one hand, the lower-level problem finds a representation by successively minimizing layer-wise objectives made of the sum of a prescribed regularizer, a fidelity term and a linear function depending on the representation found at the previous layer. On the other hand, the upper-level problem optimizes over the linear functions to yield a linearly separable final representation. We show that, by choosing the fidelity term as the quadratic distance between two successive layer-wise representations, the bilevel problem reduces to the training of a feedforward neural network. Instead, by elaborating on Bregman distances, we devise a novel neural network architecture additionally involving the inverse of the activation function reminiscent of the skip connection used in ResNets. Numerical experiments suggest that the proposed Bregman variant benefits from better learning properties and more robust prediction performance.
| Type: | Proceedings paper |
|---|---|
| Title: | Bregman Neural Networks |
| Event: | 38th International Conference on Machine Learning (ICML) |
| Location: | Baltimore, MD |
| Dates: | 17 Jul 2022 - 23 Jul 2022 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://proceedings.mlr.press/v162/ |
| Language: | English |
| Additional information: | This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third-party material in this article are included in the Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ |
| Keywords: | Science & Technology, Technology, Computer Science, Artificial Intelligence, Computer Science, DEEP |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10173882 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
