eprintid: 10198474
rev_number: 7
eprint_status: archive
userid: 699
dir: disk0/10/19/84/74
datestamp: 2024-10-15 11:48:51
lastmod: 2024-10-15 11:48:51
status_changed: 2024-10-15 11:48:51
type: article
metadata_visibility: show
sword_depositor: 699
creators_name: Shah, Shaik Basheeruddin
creators_name: Pradhan, Pradyumna
creators_name: Pu, Wei
creators_name: Randhi, Ramunaidu
creators_name: Rodrigues, Miguel RD
creators_name: Eldar, Yonina C
title: Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding
ispublished: pub
divisions: UCL
divisions: B04
divisions: F46
keywords: Optimization guarantees, 
algorithm unfolding, 
LISTA, 
ADMM-CSNet, 
Polyak-Łojasiewicz condition
note: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
abstract: Solving linear inverse problems plays a crucial role in numerous applications. Algorithm unfolding based, model-aware data-driven approaches have gained significant attention for effectively addressing these problems. Learned iterative soft-thresholding algorithm (LISTA) and alternating direction method of multipliers compressive sensing network (ADMM-CSNet) are two widely used such approaches, based on ISTA and ADMM algorithms, respectively. In this work, we study optimization guarantees, i.e., achieving near-zero training loss with the increase in the number of learning epochs, for finite-layer unfolded networks such as LISTA and ADMM-CSNet with smooth soft-thresholding in an over-parameterized (OP) regime. We achieve this by leveraging a modified version of the Polyak-Łojasiewicz, denoted PL*, condition. Satisfying the PL* condition within a specific region of the loss landscape ensures the existence of a global minimum and exponential convergence from initialization using gradient descent based methods. Hence, we provide conditions, in terms of the network width and the number of training samples, on these unfolded networks for the PL* condition to hold, by deriving the Hessian spectral norm. Additionally, we show that the threshold on the number of training samples increases with the increase in the network width. Furthermore, we compare the threshold on training samples of unfolded networks with that of a standard fully-connected feed-forward network (FFNN) with smooth soft-thresholding non-linearity. We prove that unfolded networks have a higher threshold value than FFNN. Consequently, one can expect a better expected error for unfolded networks than FFNN.
date: 2024-06-11
date_type: published
publisher: Institute of Electrical and Electronics Engineers (IEEE)
official_url: https://doi.org/10.1109/TSP.2024.3412981
oa_status: green
full_text_type: other
language: eng
primo: open
primo_central: open_green
verified: verified_manual
elements_id: 2299448
doi: 10.1109/tsp.2024.3412981
lyricists_name: Rodrigues, Miguel
lyricists_id: MRDIA06
actors_name: Rodrigues, Miguel
actors_id: MRDIA06
actors_role: owner
funding_acknowledgements: [Alan Turing Institute]; 129589 [Weizmann - UK Making Connections Programme]; 02011/26/2023/NBHM(R.P)/RDII/5867 [National Board for Higher Mathematics(NBHM), Govt. of India]
full_text_status: public
publication: IEEE Transactions on Signal Processing
volume: 72
pagerange: 3272-3286
issn: 1053-587X
citation:        Shah, Shaik Basheeruddin;    Pradhan, Pradyumna;    Pu, Wei;    Randhi, Ramunaidu;    Rodrigues, Miguel RD;    Eldar, Yonina C;      (2024)    Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding.                   IEEE Transactions on Signal Processing , 72    pp. 3272-3286.    10.1109/tsp.2024.3412981 <https://doi.org/10.1109/tsp.2024.3412981>.       Green open access   
 
document_url: https://discovery.ucl.ac.uk/id/eprint/10198474/1/2309.06195v1.pdf