[Paper Review] Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

Yu-Xuan Zhang, Wenzheng Chen+5 more|arXiv (Cornell University)|Oct 18, 2020

Generative Adversarial Networks and Image Synthesis39 references57 citations

TL;DR

The paper trains an inverse graphics network using a GAN-generated multi-view dataset and a differentiable renderer, then disentangles StyleGAN latent space to create a controllable 3D neural renderer (StyleGAN-R).

ABSTRACT

Differentiable rendering has paved the way to training neural networks to perform "inverse graphics" tasks such as predicting 3D geometry from monocular photographs. To train high performing models, most of the current approaches rely on multi-view imagery which are not readily available in practice. Recent Generative Adversarial Networks (GANs) that synthesize images, in contrast, seem to acquire 3D knowledge implicitly during training: object viewpoints can be manipulated by simply manipulating the latent codes. However, these latent codes often lack further physical interpretation and thus GANs cannot easily be inverted to perform explicit 3D reasoning. In this paper, we aim to extract and disentangle 3D knowledge learned by generative models by utilizing differentiable renderers. Key to our approach is to exploit GANs as a multi-view data generator to train an inverse graphics network using an off-the-shelf differentiable renderer, and the trained inverse graphics network as a teacher to disentangle the GAN's latent code into interpretable 3D properties. The entire architecture is trained iteratively using cycle consistency losses. We show that our approach significantly outperforms state-of-the-art inverse graphics networks trained on existing datasets, both quantitatively and via user studies. We further showcase the disentangled GAN as a controllable 3D "neural renderer", complementing traditional graphics renderers.

Motivation & Objective

• Leverage StyleGAN as a multi-view data generator to enable inverse graphics from monocular images.
• Train a differentiable renderer-based inverse graphics network to predict 3D geometry and texture.
• Disentangle StyleGAN latent space using inverse graphics to obtain an interpretable 3D representation.
• Turn StyleGAN into a controllable 3D neural renderer (StyleGAN-R) with explicit 3D control over shape, texture, and background.
• Show improved 3D reconstruction quality and demonstrate 3D manipulation and neural rendering capabilities.

Proposed method

• Use StyleGAN to generate multi-view images by fixing a viewpoint code w_v* and varying content codes.
• Train an inverse graphics network f_theta with DIB-R as the differentiable renderer to predict mesh S and texture T from images I_V and masks M.
• Apply multi-view consistency loss across pairs of views to supervise 3D properties.
• Train a mapping network that maps viewpoint, shape, texture, and background to StyleGAN’s latent space W*, enabling disentanglement (StyleGAN-R).
• Fine-tune StyleGAN jointly with the mapping network using cycle-consistency losses to separate foreground from background and encourage disentanglement.
• Optionally render results using StyleGAN-R to compare with a traditional differentiable renderer (DIB-R).

Experimental results

Research questions

• RQ1 Can GANs learn 3D knowledge implicitly and be made invertible for explicit 3D reasoning without 3D ground-truth data?
• RQ2 Does a differentiable renderer-based inverse graphics network trained on GAN-generated multi-view data outperform models trained on conventional 3D datasets?
• RQ3 Can the latent space of StyleGAN be disentangled into interpretable 3D properties (shape, texture, background) via a learned mapping from 3D properties?
• RQ4 Does disentangling GANs enable a controllable 3D neural renderer capable of viewpoint, shape, texture, and background manipulation?
• RQ5 What is the impact of multi-view consistency losses on reconstruction quality?

Key findings

• Inverse graphics trained on the StyleGAN multi-view dataset significantly improves 3D reconstruction quality compared with models trained on Pascal3D.
• The StyleGAN-generated dataset enables much larger and efficiently annotated training data (e.g., 50K StyleGAN cars vs. ~4K Pascal3D), with annotation time dramatically reduced (minutes versus hundreds of hours).
• The learned mapping network and fine-tuned StyleGAN produce a disentangled StyleGAN-R capable of controllable 3D rendering, including viewpoint changes and 3D property swaps (shape, texture, background).
• Experiments show higher 2D IOU scores when evaluated on re-projected views and user studies favor StyleGAN-based reconstructions over the Pascal3D baseline.
• Ablation studies demonstrate the importance of multi-view consistency loss for improving texture and shape quality in predictions.