Pandey, Karran;
Guerrero, Paul;
Gadelha, Matheus;
Hold-Geoffroy, Yannick;
Singh, Karan;
Mitra, Niloy J;
(2024)
Diffusion Handles Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D.
In:
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024.
(pp. pp. 7695-7704).
Institute of Electrical and Electronics Engineers (IEEE)
Preview |
Text
Pandey_Diffusion_Handles_Enabling_3D_Edits_for_Diffusion_Models_by_Lifting_CVPR_2024_paper.pdf - Accepted Version Download (10MB) | Preview |
Abstract
Diffusion Handles is a novel approach to enable 3D object edits on diffusion images, requiring only existing pre-trained diffusion models depth estimation, without any fine-tuning or 3D object retrieval. The edited results remain plausible, photo-real, and preserve object identity. Diffusion Handles address a critically missing facet of generative image-based creative design. Our key insight is to lift diffusion activations for a selected object to 3D using a proxy depth, 3D-transform the depth and associated activations, and project them back to image space. The diffusion process guided by the manipulated activations produces plausible edited images showing complex 3D occlusion and lighting effects. We evaluate Diffusion Handles: quantitatively, on a large synthetic data benchmark; and qualitatively by a user study, showing our output to be more plausible, and better than prior art at both, 3D editing and identity control.
| Type: | Proceedings paper |
|---|---|
| Title: | Diffusion Handles Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D |
| Event: | IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) |
| Location: | Seattle, WA, USA |
| Dates: | 16th-22nd June 2024 |
| ISBN-13: | 979-8-3503-5301-3 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1109/CVPR52733.2024.00735 |
| Publisher version: | https://doi.org/10.1109/cvpr52733.2024.00735 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher's terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10203447 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
