Denker, Alexander;
Vargas, Francisco;
Padhy, Shreyas;
Didi, Kieran;
Mathis, Simon V;
Barbano, Riccardo;
Dutordoir, Vincent;
... Lio, Pietro; + view all
(2024)
DEFT: Efficient Fine-tuning of Diffusion Models by Learning the Generalised h-transform.
In:
Proceedings of the Thirty-eighth Annual Conference on Neural Information Processing Systems.
(pp. pp. 1-47).
NeurIPS
Preview |
Text
11591_DEFT_Efficient_Fine_tuni-2.pdf - Published Version Download (13MB) | Preview |
Abstract
Generative modelling paradigms based on denoising diffusion processes have emerged as a leading candidate for conditional sampling in inverse problems. In many real-world applications, we often have access to large, expensively trained unconditional diffusion models, which we aim to exploit for improving conditional sampling. Most recent approaches are motivated heuristically and lack a unifying framework, obscuring connections between them. Further, they often suffer from issues such as being very sensitive to hyperparameters, being expensive to train or needing access to weights hidden behind a closed API. In this work, we unify conditional training and sampling using the mathematically well-understood Doob's h-transform. This new perspective allows us to unify many existing methods under a common umbrella. Under this framework, we propose DEFT (Doob's h-transform Efficient FineTuning), a new approach for conditional generation that simply fine-tunes a very small network to quickly learn the conditional h-transform, while keeping the larger unconditional network unchanged. DEFT is much faster than existing baselines while achieving state-of-the-art performance across a variety of linear and non-linear benchmarks. On image reconstruction tasks, we achieve speedups of up to 1.6x, while having the best perceptual quality on natural images and reconstruction performance on medical images. Further, we also provide initial experiments on protein motif scaffolding and outperform reconstruction guidance methods.
| Type: | Proceedings paper |
|---|---|
| Title: | DEFT: Efficient Fine-tuning of Diffusion Models by Learning the Generalised h-transform |
| Event: | NeurIPS 2024 |
| Location: | Vancouver, Canada |
| Dates: | 10th-15th December 2024 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://openreview.net/forum?id=AKBTFQhCjm |
| Language: | English |
| Additional information: | © The Authors 2024. Original content in this paper is licensed under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0) Licence (https://creativecommons.org/licenses/by/4.0/). |
| Keywords: | Inverse problems, Generative Modelling, Diffusion Models, Conditional Generative Modelling, Diffusion model guidance |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10204141 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
