[Paper Review] Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Jiajun Wu, Chengkai Zhang+3 more|arXiv (Cornell University)|Oct 24, 2016

Image Processing and 3D Reconstruction1,562 citations

TL;DR

This paper introduces 3D-GAN to generate high-quality 3D voxel objects from a latent space, and extends it with 3D-VAE-GAN to map 2D images to 3D objects, enabling unsupervised shape learning and single-image reconstruction.

ABSTRACT

We study the problem of 3D object generation. We propose a novel framework, namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets. The benefits of our model are three-fold: first, the use of an adversarial criterion, instead of traditional heuristic criteria, enables the generator to capture object structure implicitly and to synthesize high-quality 3D objects; second, the generator establishes a mapping from a low-dimensional probabilistic space to the space of 3D objects, so that we can sample objects without a reference image or CAD models, and explore the 3D object manifold; third, the adversarial discriminator provides a powerful 3D shape descriptor which, learned without supervision, has wide applications in 3D object recognition. Experiments demonstrate that our method generates high-quality 3D objects, and our unsupervisedly learned features achieve impressive performance on 3D object recognition, comparable with those of supervised learning methods.

Motivation & Objective

• Motivate learning a generative model of 3D object shapes that yields varied and realistic outputs.
• Propose a 3D-GAN framework that maps latent vectors to 64^3 voxel objects using volumetric CNNs and adversarial training.
• Demonstrate that the discriminator learns useful unsupervised 3D shape representations for recognition.
• Extend to 3D-VAE-GAN to enable reconstruction from 2D images and latent-space mapping from images to 3D shapes.

Proposed method

• Introduce 3D-GAN: a generator G maps a 200-dim latent vector z to a 64×64×64 voxel object; a discriminator D scores real vs. generated objects using an all-convolutional network.
• Train with an adversarial loss L3D-GAN = log D(x) + log(1 − D(G(z))).
• Adopt adaptive training where the discriminator updates only if its accuracy in the last batch is ≤ 80% to stabilize training.
• Extend to 3D-VAE-GAN by adding an image encoder E that maps 2D images to latent z, and optimize a combined loss L = L3D-GAN + α1 L_KL + α2 L_recon.
• L_KL enforces q(z|y) to follow a prior p(z); L_recon minimizes ||G(E(y)) − x||2 for 3D reconstruction.
• Render training 3D shapes from SUN indoor images across multiple viewpoints for 3D-VAE-GAN training.

Experimental results

Research questions

• RQ1 Can a 3D-GAN learn a probabilistic latent space that generates novel, high-quality 3D objects without explicit part-based modeling?
• RQ2 Do adversarially learned 3D representations provide competitive unsupervised features for 3D shape classification?
• RQ3 Can a 3D-VAE-GAN map 2D images to corresponding 3D shapes enabling single-image 3D reconstruction?
• RQ4 What semantic structure exists in the learned latent space and discriminator activations that reflects object parts and categories?

Key findings

• The 3D-GAN generates high-quality, high-resolution 3D objects from latent vectors, exceeding prior probabilistic methods.
• The discriminator yields unsupervised yet effective 3D shape features that achieve competitive 3D object classification performance, outperforming several other unsupervised methods.
• The 3D-VAE-GAN enables reconstructing 3D objects from 2D images and demonstrates shape arithmetic and interpolation in latent space.
• Latent-space analyses reveal semantically meaningful dimensions, smooth interpolations across and within categories, and neuron activations corresponding to coherent object parts.