[Paper Review] Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data

Sebastian Lunz, Yingzhen Li+2 more|arXiv (Cornell University)|Feb 28, 2020

3D Shape Modeling and Analysis60 references37 citations

TL;DR

IG-GAN learns to generate 3D shapes from unstructured 2D images by using an off-the-shelf non-differentiable renderer paired with a proxy neural renderer and a discriminator output matching loss to propagate gradients through the rendering process.

ABSTRACT

Recent work has shown the ability to learn generative models for 3D shapes from only unstructured 2D images. However, training such models requires differentiating through the rasterization step of the rendering process, therefore past work has focused on developing bespoke rendering models which smooth over this non-differentiable process in various ways. Such models are thus unable to take advantage of the photo-realistic, fully featured, industrial renderers built by the gaming and graphics industry. In this paper we introduce the first scalable training technique for 3D generative models from 2D data which utilizes an off-the-shelf non-differentiable renderer. To account for the non-differentiability, we introduce a proxy neural renderer to match the output of the non-differentiable renderer. We further propose discriminator output matching to ensure that the neural renderer learns to smooth over the rasterization appropriately. We evaluate our model on images rendered from our generated 3D shapes, and show that our model can consistently learn to generate better shapes than existing models when trained with exclusively unstructured 2D images.

Motivation & Objective

• Enable 3D shape generation trained solely from unstructured 2D images without differentiable rendering constraints.
• Leverage off-the-shelf industry renderers by introducing a trainable neural proxy renderer.
• Patch non-differentiability via discriminator output matching to align continuous voxel representations with discrete renderings.
• Demonstrate improved 3D shape generation quality over prior 2D-supervised methods across synthetic and natural datasets.

Proposed method

• Represent 3D shapes as voxels with continuous occupancy values.
• Use an off-the-shelf non-differentiable renderer to render discrete voxel thresholds.
• Introduce a proxy neural renderer that maps continuous voxels to 2D images and train it to match the off-the-shelf renderer on discrete inputs (L2 loss).
• Define a discriminator output matching (DOM) loss to align the neural renderer’s outputs with the discriminator’s expectations when discrete renders are thresholded.
• Train with a GAN objective where gradients flow from the discriminator through the neural renderer back to the 3D generator via the proxy renderer.
• Employ a reward-approximation step that uses the neural renderer to provide differentiable feedback for the continuous voxel generator (bypassing intractable score functions).

Experimental results

Research questions

• RQ1 Can unstructured 2D images be used to learn a scalable 3D shape generator without differentiable rendering?
• RQ2 Does a proxy neural renderer paired with DOM loss enable effective gradient flow through non-differentiable rendering steps?
• RQ3 How does IG-GAN compare to existing 2D-supervised 3D generation methods on synthetic and natural datasets?
• RQ4 To what extent do lighting, shading, and concavities in 2D images improve the quality of generated 3D shapes?

Key findings

• IG-GAN consistently improves 2D FID scores over baselines on ShapeNet-derived datasets (bathtubs, couches, chairs).
• The method especially excels on concave objects (e.g., bathtubs and couches) due to better use of lighting and shading cues.
• Discriminator Output Matching (DOM) is crucial for training stability and quality; removing or weakening DOM degrades results.
• The approach yields realistic 3D shapes when trained on natural images (chanterelle mushrooms) as well as synthetic data.
• Pre-training the neural renderer is beneficial, but robustness to different pre-trainings is observed, provided domain-relevant data is used for pre-training.