Marinescu, Razvan V;
Moyer, Daniel;
Golland, Polina;
(2021)
Bayesian Image Reconstruction using Deep
Generative Models.
In:
Proceedings of NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.
NeurIPS (Neural Information Processing Systems Foundation): San Diego, CA, USA.
Preview |
Text
Marinescu_Bayesian Image Reconstruction using Deep Generative Models..pdf Download (20MB) | Preview |
Abstract
Machine learning models are commonly trained end-to-end and in a supervised setting, using paired (input, output) data. Examples include recent super-resolution methods that train on pairs of (low-resolution, high-resolution) images. However, these end-to-end approaches require re-training every time there is a distribution shift in the inputs (e.g., night images vs daylight) or relevant latent variables (e.g., camera blur or hand motion). In this work, we leverage state-of-the-art (SOTA) generative models (here StyleGAN2) for building powerful image priors, which enable application of Bayes’ theorem for many downstream reconstruction tasks. Our method, Bayesian Reconstruction through Generative Models (BRGM), uses a single pre-trained generator model to solve different image restoration tasks, i.e., super-resolution and in-painting, by combining it with different forward corruption models. We keep the weights of the generator model fixed, and reconstruct the image by estimating the Bayesian maximum a-posteriori (MAP) estimate over the input latent vector that generated the reconstructed image. We further use Variational Inference to approximate the posterior distribution over the latent vectors, from which we sample multiple solutions. We demonstrate BRGM on three large and diverse datasets: (i) 60,000 images from the Flick Faces High Quality dataset [1] (ii) 240,000 chest X-rays from MIMIC III [2] and (iii) a combined collection of 5 brain MRI datasets with 7,329 scans [3]. Across all three datasets and without any dataset-specific hyperparameter tuning, our simple approach yields performance competitive with current task-specific state-of-the-art methods on super-resolution and in-painting, while being more generalisable and without requiring any training. Our source code and pre-trained models are available online: https://razvanmarinescu.github.io/brgm/
| Type: | Proceedings paper |
|---|---|
| Title: | Bayesian Image Reconstruction using Deep Generative Models |
| Event: | Deep Generative Models and Downstream Applications Workshop |
| Dates: | 14 Dec 2021 - 14 Dec 2021 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://openreview.net/forum?id=P5y8Ux34Exj |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Image Reconstruction, Deep Generative Models, Bayesian Models, Deep Generative Priors |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10176403 |
Archive Staff Only
 |
View Item |
