Jin, B;
Lu, X;
(2019)
On the regularizing property of stochastic gradient descent.
Inverse Problems
, 35
(1)
, Article 015004. 10.1088/1361-6420/aaea2a.
Preview |
Text
Jin_On the Regularizing Property of Stochastic Gradient Descent_VoR.pdf - Published Version Download (7MB) | Preview |
Abstract
Stochastic gradient descent (SGD) and its variants are among the most successful approaches for solving large-scale optimization problems. At each iteration, SGD employs an unbiased estimator of the full gradient computed from one single randomly selected data point. Hence, it scales well with problem size and is very attractive for handling truly massive dataset, and holds significant potentials for solving large-scale inverse problems. In this work, we rigorously establish its regularizing property under a priori early stopping rule for linear inverse problems, and also prove convergence rates under the canonical sourcewise condition. This is achieved by combining tools from classical regularization theory and stochastic analysis. Further, we analyze its preasymptotic weak and strong convergence behavior, in order to explain the fast initial convergence typically observed in practice. The theoretical findings shed insights into the performance of the algorithm, and are complemented with illustrative numerical experiments.
| Type: | Article |
|---|---|
| Title: | On the regularizing property of stochastic gradient descent |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1088/1361-6420/aaea2a |
| Publisher version: | https://doi.org/10.1088/1361-6420/aaea2a |
| Language: | English |
| Additional information: | © 2018 IOP Publishing Ltd. Original content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence (http://creativecommons.org/licenses/by/3.0). |
| Keywords: | stochastic gradient descent, regularizing property, error estimates, preasymptotic convergence |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10059340 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
