[Paper Review] Neural Graphics Pipeline for Controllable Image Generation

Xuelin Chen, Daniel Cohen‐Or+2 more|arXiv (Cornell University)|Jun 18, 2020

Computer Graphics and Visualization Techniques36 references4 citations

TL;DR

Neural Graphics Pipeline (NGP) is a hybrid generative model that combines neural and traditional image formation to enable controllable image generation. It generates coarse 3D models, renders view-specific 2D maps via neural rendering, and composites them using traditional rendering, achieving improved FID scores and direct control over illumination, camera, shape, and appearance without explicit image correspondence during unsupervised training.

ABSTRACT

We present Neural Graphics Pipeline (NGP), a hybrid generative model that brings together neural and traditional image formation models. NGP generates coarse 3D models that are fed into neural rendering modules to produce view-specific interpretable 2D maps, which are then composited into the final output image using a traditional image formation model. Our approach offers control over image generation by providing direct handles controlling illumination and camera parameters, in addition to control over shape and appearance variations. The key challenge is to learn these controls through unsupervised training that links generated coarse 3D models with unpaired real images via neural and traditional (e.g., Blinn-Phong) rendering functions without establishing an explicit correspondence between them. We evaluate our hybrid modeling framework, compare with neural-only generation methods (namely, DCGAN, LSGAN, WGAN-GP, VON, and SRNs), report improvement in FID scores against real images, and demonstrate that NGP supports direct controls common in traditional forward rendering. Code, data, and trained models will be released on acceptance.

Motivation & Objective

• To develop a generative model that enables fine-grained control over image generation beyond shape and appearance.
• To bridge neural and traditional image formation models for improved controllability and realism.
• To train the model in an unsupervised manner without requiring explicit correspondence between generated 3D shapes and real images.
• To support direct manipulation of illumination and camera parameters during image generation, as in traditional rendering pipelines.
• To achieve state-of-the-art performance in image generation quality, measured by FID scores, compared to neural-only methods.

Proposed method

• NGP generates coarse 3D models as the initial latent representation for image synthesis.
• Neural rendering modules produce view-specific 2D feature maps conditioned on the 3D geometry and view parameters.
• Traditional image formation models, such as Blinn-Phong, are used to composite the rendered 2D maps into the final image.
• The model is trained in an unsupervised manner by aligning generated images with real images via differentiable neural and traditional rendering functions.
• Control signals for illumination and camera parameters are directly injected into the rendering pipeline, enabling end-to-end controllability.
• The framework avoids explicit correspondence between generated 3D shapes and real images by leveraging differentiable rendering and adversarial training.

Experimental results

Research questions

• RQ1 Can a hybrid neural-traditional rendering pipeline enable controllable image generation with direct manipulation of lighting and camera parameters?
• RQ2 How well can a model trained without explicit correspondence between generated 3D shapes and real images perform in image generation compared to purely neural methods?
• RQ3 To what extent does combining neural and traditional rendering improve image quality and controllability in generative models?
• RQ4 Can the NGP framework support the same level of control as traditional forward rendering pipelines while maintaining generative flexibility?
• RQ5 What is the performance gain of NGP over neural-only generative models like DCGAN, WGAN-GP, and SRNs in terms of FID scores?

Key findings

• NGP achieves improved Fréchet Inception Distance (FID) scores compared to neural-only generative models such as DCGAN, LSGAN, WGAN-GP, VON, and SRNs.
• The model supports direct control over illumination and camera parameters, enabling manipulation similar to traditional rendering pipelines.
• The unsupervised training scheme successfully links coarse 3D models to real images without requiring explicit 2D-3D correspondence.
• The hybrid architecture enables higher image quality and better disentanglement of shape, appearance, lighting, and camera control.
• The framework demonstrates that combining neural and traditional rendering improves both controllability and generation fidelity.
• Code, data, and trained models will be released upon acceptance, supporting reproducibility and further research.