TY  - JOUR
SP  - 127
VL  - 40
IS  - 2
N1  - This version is the author accepted manuscript. For information on re-use, please refer to the publisher?s terms and conditions.
UR  - https://doi.org/10.1111/cgf.142620
JF  - Computer Graphics Forum
A1  - Chen, X
A1  - Cohen-Or, D
A1  - Chen, B
A1  - Mitra, NJ
TI  - Towards a Neural Graphics Pipeline for Controllable Image Generation
AV  - public
Y1  - 2021/05/01/
EP  - 140
N2  - In this paper, we leverage advances in neural networks towards forming a neural rendering for controllable image generation, and thereby bypassing the need for detailed modeling in conventional graphics pipeline. To this end, we present Neural Graphics Pipeline (NGP), a hybrid generative model that brings together neural and traditional image formation models. NGP decomposes the image into a set of interpretable appearance feature maps, uncovering direct control handles for controllable image generation. To form an image, NGP generates coarse 3D models that are fed into neural rendering modules to produce view-specific interpretable 2D maps, which are then composited into the final output image using a traditional image formation model. Our approach offers control over image generation by providing direct handles controlling illumination and camera parameters, in addition to control over shape and appearance variations. The key challenge is to learn these controls through unsupervised training that links generated coarse 3D models with unpaired real images via neural and traditional (e.g., Blinn-Phong) rendering functions, without establishing an explicit correspondence between them. We demonstrate the effectiveness of our approach on controllable image generation of single-object scenes. We evaluate our hybrid modeling framework, compare with neural-only generation methods (namely, DCGAN, LSGAN, WGAN-GP, VON, and SRNs), report improvement in FID scores against real images, and demonstrate that NGP supports direct controls common in traditional forward rendering. Code is available at http://geometry.cs.ucl.ac.uk/projects/2021/ngp.
ID  - discovery10139850
PB  - WILEY
KW  - CCS Concepts
? Computing methodologies ? Rendering; Shape modeling;
ER  -