UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

On the regularizing property of stochastic gradient descent

Jin, B; Lu, X; (2019) On the regularizing property of stochastic gradient descent. Inverse Problems , 35 (1) , Article 015004. 10.1088/1361-6420/aaea2a. Green open access

[thumbnail of Jin_On the Regularizing Property of Stochastic Gradient Descent_VoR.pdf]
Preview
Text
Jin_On the Regularizing Property of Stochastic Gradient Descent_VoR.pdf - Published Version

Download (7MB) | Preview

Abstract

Stochastic gradient descent (SGD) and its variants are among the most successful approaches for solving large-scale optimization problems. At each iteration, SGD employs an unbiased estimator of the full gradient computed from one single randomly selected data point. Hence, it scales well with problem size and is very attractive for handling truly massive dataset, and holds significant potentials for solving large-scale inverse problems. In this work, we rigorously establish its regularizing property under a priori early stopping rule for linear inverse problems, and also prove convergence rates under the canonical sourcewise condition. This is achieved by combining tools from classical regularization theory and stochastic analysis. Further, we analyze its preasymptotic weak and strong convergence behavior, in order to explain the fast initial convergence typically observed in practice. The theoretical findings shed insights into the performance of the algorithm, and are complemented with illustrative numerical experiments.

Type: Article
Title: On the regularizing property of stochastic gradient descent
Open access status: An open access version is available from UCL Discovery
DOI: 10.1088/1361-6420/aaea2a
Publisher version: https://doi.org/10.1088/1361-6420/aaea2a
Language: English
Additional information: © 2018 IOP Publishing Ltd. Original content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence (http://creativecommons.org/licenses/by/3.0).
Keywords: stochastic gradient descent, regularizing property, error estimates, preasymptotic convergence
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10059340
Downloads since deposit
115Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item