UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding

Shah, Shaik Basheeruddin; Pradhan, Pradyumna; Pu, Wei; Randhi, Ramunaidu; Rodrigues, Miguel RD; Eldar, Yonina C; (2024) Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding. IEEE Transactions on Signal Processing , 72 pp. 3272-3286. 10.1109/tsp.2024.3412981. Green open access

[thumbnail of 2309.06195v1.pdf]
Preview
Text
2309.06195v1.pdf - Accepted Version

Download (955kB) | Preview

Abstract

Solving linear inverse problems plays a crucial role in numerous applications. Algorithm unfolding based, model-aware data-driven approaches have gained significant attention for effectively addressing these problems. Learned iterative soft-thresholding algorithm (LISTA) and alternating direction method of multipliers compressive sensing network (ADMM-CSNet) are two widely used such approaches, based on ISTA and ADMM algorithms, respectively. In this work, we study optimization guarantees, i.e., achieving near-zero training loss with the increase in the number of learning epochs, for finite-layer unfolded networks such as LISTA and ADMM-CSNet with smooth soft-thresholding in an over-parameterized (OP) regime. We achieve this by leveraging a modified version of the Polyak-Łojasiewicz, denoted PL*, condition. Satisfying the PL* condition within a specific region of the loss landscape ensures the existence of a global minimum and exponential convergence from initialization using gradient descent based methods. Hence, we provide conditions, in terms of the network width and the number of training samples, on these unfolded networks for the PL* condition to hold, by deriving the Hessian spectral norm. Additionally, we show that the threshold on the number of training samples increases with the increase in the network width. Furthermore, we compare the threshold on training samples of unfolded networks with that of a standard fully-connected feed-forward network (FFNN) with smooth soft-thresholding non-linearity. We prove that unfolded networks have a higher threshold value than FFNN. Consequently, one can expect a better expected error for unfolded networks than FFNN.

Type: Article
Title: Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding
Open access status: An open access version is available from UCL Discovery
DOI: 10.1109/tsp.2024.3412981
Publisher version: https://doi.org/10.1109/TSP.2024.3412981
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: Optimization guarantees, algorithm unfolding, LISTA, ADMM-CSNet, Polyak-Łojasiewicz condition
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Electronic and Electrical Eng
URI: https://discovery.ucl.ac.uk/id/eprint/10198474
Downloads since deposit
Loading...
13Downloads
Download activity - last month
Loading...
Download activity - last 12 months
Loading...
Downloads by country - last 12 months
Loading...

Archive Staff Only

View Item View Item