Chen, X;
Cohen-Or, D;
Chen, B;
Mitra, NJ;
(2021)
Towards a Neural Graphics Pipeline for Controllable Image Generation.
Computer Graphics Forum
, 40
(2)
pp. 127-140.
10.1111/cgf.142620.
Preview |
Text
neuralGraphics.pdf - Accepted Version Download (18MB) | Preview |
Abstract
In this paper, we leverage advances in neural networks towards forming a neural rendering for controllable image generation, and thereby bypassing the need for detailed modeling in conventional graphics pipeline. To this end, we present Neural Graphics Pipeline (NGP), a hybrid generative model that brings together neural and traditional image formation models. NGP decomposes the image into a set of interpretable appearance feature maps, uncovering direct control handles for controllable image generation. To form an image, NGP generates coarse 3D models that are fed into neural rendering modules to produce view-specific interpretable 2D maps, which are then composited into the final output image using a traditional image formation model. Our approach offers control over image generation by providing direct handles controlling illumination and camera parameters, in addition to control over shape and appearance variations. The key challenge is to learn these controls through unsupervised training that links generated coarse 3D models with unpaired real images via neural and traditional (e.g., Blinn-Phong) rendering functions, without establishing an explicit correspondence between them. We demonstrate the effectiveness of our approach on controllable image generation of single-object scenes. We evaluate our hybrid modeling framework, compare with neural-only generation methods (namely, DCGAN, LSGAN, WGAN-GP, VON, and SRNs), report improvement in FID scores against real images, and demonstrate that NGP supports direct controls common in traditional forward rendering. Code is available at http://geometry.cs.ucl.ac.uk/projects/2021/ngp.
| Type: | Article |
|---|---|
| Title: | Towards a Neural Graphics Pipeline for Controllable Image Generation |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1111/cgf.142620 |
| Publisher version: | https://doi.org/10.1111/cgf.142620 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | CCS Concepts • Computing methodologies → Rendering; Shape modeling; |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10139850 |
Archive Staff Only
 |
View Item |
