Glaser, Pierre;
Huang, Kevin Han;
Gretton, Arthur;
(2024)
Near-Optimality of Contrastive Divergence Algorithms.
In: Globerson, A and Mackey, L and Belgrave, D and Fan, A and Paquet, U and Tomczak, J and Zhang, C, (eds.)
Advances in Neural Information Processing Systems 37 (NeurIPS 2024).
(pp. pp. 1-55).
NeurIPS: Vancouver, BC, Canada.
Preview |
Text
NeurIPS-2024-near-optimality-of-contrastive-divergence-algorithms-Paper-Conference.pdf - Published Version Download (667kB) | Preview |
Abstract
We provide a non-asymptotic analysis of the contrastive divergence (CD) algorithm, a training method for unnormalized models. While prior work has established that (for exponential family distributions) the CD iterates asymptotically converge at an O ( n − 1 / 3 ) rate to the true parameter of the data distribution, we show that CD can achieve the parametric rate O ( n − 1 / 2 ) . Our analysis provides results for various data batching schemes, including fully online and minibatch. We additionally show that CD is near-optimal, in the sense that its asymptotic variance is close to the Cramér-Rao lower bound.
| Type: | Proceedings paper |
|---|---|
| Title: | Near-Optimality of Contrastive Divergence Algorithms |
| Event: | 38th Conference on Neural Information Processing Systems (NeurIPS 2024). |
| ISBN-13: | 9798331314385 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://papers.nips.cc/paper_files/paper/2024/hash... |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Gatsby Computational Neurosci Unit |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10207214 |
Archive Staff Only
 |
View Item |
