Sarao Mannelli, Stefano;
Biroli, Giulio;
Cammarota, Chiara;
Krzakala, Florent;
Urbani, Pierfrancesco;
Zdeborova, Lenka;
(2020)
Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval.
In:
NeurIPS Proceedings.
NeurIPS
Preview |
Text
NeurIPS-2020-complex-dynamics-in-simple-neural-networks-understanding-gradient-flow-in-phase-retrieval-Paper.pdf - Published Version Download (616kB) | Preview |
Abstract
Despite the widespread use of gradient-based algorithms for optimising high-dimensional non-convex functions, understanding their ability of finding good minima instead of being trapped in spurious ones remains to a large extent an open problem. Here we focus on gradient flow dynamics for phase retrieval from random measurements. When the ratio of the number of measurements over the input dimension is small the dynamics remains trapped in spurious minima with large basins of attraction. We find analytically that above a critical ratio those critical points become unstable developing a negative direction toward the signal. By numerical experiments we show that in this regime the gradient flow algorithm is not trapped; it drifts away from the spurious critical points along the unstable direction and succeeds in finding the global minimum. Using tools from statistical physics we characterise this phenomenon, which is related to a BBP-type transition in the Hessian of the spurious minima.
| Type: | Proceedings paper |
|---|---|
| Title: | Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval |
| Event: | Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://proceedings.neurips.cc/paper/2020/hash/217... |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Gatsby Computational Neurosci Unit |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10166776 |
Archive Staff Only
 |
View Item |
