[Paper Review] Unsupervised Learning of 3D Structure from Images

Danilo Jimenez Rezende, S. M. Ali Eslami+4 more|arXiv (Cornell University)|Jul 3, 2016

Advanced Vision and Imaging32 references97 citations

TL;DR

This paper proposes an unsupervised deep generative model that learns 3D structure from 2D images using differentiable rendering and end-to-end training, achieving state-of-the-art log-likelihoods on ShapeNet and MNIST3D while enabling multi-modal 3D reconstruction from single or multiple views without any 3D supervision.

ABSTRACT

A key goal of computer vision is to recover the underlying 3D structure from 2D observations of the world. In this paper we learn strong deep generative models of 3D structures, and recover these structures from 3D and 2D images via probabilistic inference. We demonstrate high-quality samples and report log-likelihoods on several datasets, including ShapeNet [2], and establish the first benchmarks in the literature. We also show how these models and their inference networks can be trained end-to-end from 2D images. This demonstrates for the first time the feasibility of learning to infer 3D representations of the world in a purely unsupervised manner.

Motivation & Objective

• To learn 3D representations of objects directly from 2D images without any ground-truth 3D annotations.
• To address the ill-posed nature of 3D reconstruction by learning statistical models of plausible 3D structures.
• To enable probabilistic inference over 3D structures given 2D observations, capturing multi-modality and uncertainty.
• To demonstrate end-to-end training of 3D generative models and inference networks using only 2D images and differentiable rendering.
• To establish the first quantitative benchmarks for 3D density modeling using log-likelihoods on ShapeNet and MNIST3D.

Proposed method

• Learns a deep generative model of 3D structures over volumetric grids and meshes using latent variable models inspired by DRAW and other normalizing flows.
• Uses differentiable neural renderers (e.g., OpenGL-based) to map 3D representations to 2D image observations, enabling backpropagation through the rendering process.
• Employs conditional training to infer 3D structure from 2D images by conditioning the model on image observations and optimizing for reconstruction.
• Trains the generative model and inference network jointly in an end-to-end fashion using only 2D images, without any 3D supervision.
• Supports both volumetric and mesh-based 3D representations, with meshes parameterized by vertices constrained to move along fixed lines from the center.
• Incorporates off-the-shelf rendering engines to model realistic lighting and material interactions, improving fidelity of generated 2D views.

Experimental results

Research questions

• RQ1 Can 3D structure be inferred from 2D images in a purely unsupervised manner, without any 3D ground-truth labels?
• RQ2 How well can a deep generative model capture the multi-modal posterior distribution over 3D structures given a single 2D image?
• RQ3 What is the performance of such a model in terms of log-likelihood on standard 3D benchmark datasets like ShapeNet and MNIST3D?
• RQ4 Can differentiable rendering enable end-to-end training of 3D generative models directly from 2D images?
• RQ5 How accurately can the model reconstruct unseen 3D views of an object, demonstrating true 3D understanding beyond 2D reconstruction?

Key findings

• The model achieves state-of-the-art log-likelihood scores on both ShapeNet and MNIST3D, establishing the first quantitative benchmarks for 3D density modeling.
• The model generates high-quality, diverse 3D samples that capture the multi-modality of the data distribution, including plausible variations in shape and pose.
• Reconstructed 3D meshes generalize to unseen camera angles, accurately inferring object extents not visible in the input image, demonstrating true 3D understanding.
• The model reconstructs 2D images with high fidelity and can generate consistent 3D representations from multiple 2D views, even when trained only on 2D data.
• End-to-end training via differentiable rendering enables accurate 3D inference in a single forward pass, with the model capturing uncertainty and ambiguity in 3D structure.
• The approach successfully learns 3D representations from 2D images alone, demonstrating the feasibility of unsupervised 3D structure inference without any 3D supervision.