Kim, Seunghoi;
Jin, Chen;
Diethe, Tom;
Figini, Matteo;
Tregidgo, Henry TF;
Mullokandov, Asher;
Teare, Philip;
(2024)
Tackling Structural Hallucination in Image Translation with Local Diffusion.
In: Leonardis, Aleš and Ricci, Elisa and Roth, Stefan and Russakovsky, Olga and Sattler, Torsten and Varol, Gül, (eds.)
Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LXXXI.
(pp. pp. 87-103).
Springer: Cham, Switzerland.
![[thumbnail of eccv.pdf]](https://discovery.ucl.ac.uk/style/images/fileicons/text.png) |
Text
eccv.pdf - Published Version Access restricted to UCL open access staff until 2 November 2025. Download (3MB) |
Abstract
Recent developments in diffusion models have advanced conditioned image generation, yet they struggle with reconstructing out-of-distribution (OOD) images, such as unseen tumors in medical images, causing “image hallucination” and risking misdiagnosis. We hypothesize such hallucinations result from local OOD regions in the conditional images. We verify that partitioning the OOD region and conducting separate image generations alleviates hallucinations in several applications. From this, we propose a training-free diffusion framework that reduces hallucination with multiple Local Diffusion processes. Our approach involves OOD estimation followed by two modules: a “branching” module generates locally both within and outside OOD regions, and a “fusion” module integrates these predictions into one. Our evaluation shows our method mitigates hallucination over baseline models quantitatively and qualitatively, reducing misdiagnosis by 40% and 25% in the real-world medical and natural image datasets, respectively. It also demonstrates compatibility with various pre-trained diffusion models. Code is available at https://github.com/edshkim98/LocalDiffusion-Hallucination.
| Type: | Proceedings paper |
|---|---|
| Title: | Tackling Structural Hallucination in Image Translation with Local Diffusion |
| Event: | 18th European Conference on Computer Vision (ECCV 2024) |
| ISBN-13: | 978-3-031-73003-0 |
| DOI: | 10.1007/978-3-031-73004-7_6 |
| Publisher version: | https://doi.org/10.1007/978-3-031-73004-7_6 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Diffusion model, out-of-distribution generalization, image translation |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Med Phys and Biomedical Eng |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10201190 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
