[Paper Review] Deep Convolutional Inverse Graphics Network

Tejas D. Kulkarni, WILLIAM F. WHITNEY+2 more|arXiv (Cornell University)|Mar 11, 2015

Advanced Vision and Imaging23 references751 citations

TL;DR

This paper proposes the Deep Convolutional Inverse Graphics Network (DC-IGN), a variational autoencoder with deep convolutional and de-convolutional layers trained via Stochastic Gradient Variational Bayes to learn a disentangled, interpretable graphics code from single images. The model successfully reconstructs novel views of faces and chairs by manipulating specific latents for pose and lighting, demonstrating generalization to unseen objects and viewpoints with minimal supervision.

ABSTRACT

This paper presents the Deep Convolution Inverse Graphics Network (DC-IGN), a model that learns an interpretable representation of images. This representation is disentangled with respect to transformations such as out-of-plane rotations and lighting variations. The DC-IGN model is composed of multiple layers of convolution and de-convolution operators and is trained using the Stochastic Gradient Variational Bayes (SGVB) algorithm. We propose a training procedure to encourage neurons in the graphics code layer to represent a specific transformation (e.g. pose or light). Given a single input image, our model can generate new images of the same object with variations in pose and lighting. We present qualitative and quantitative results of the model's efficacy at learning a 3D rendering engine.

Motivation & Objective

• To learn an interpretable, disentangled representation of images that separates factors like pose, lighting, and shape.
• To train a deep generative model that can reconstruct images and synthesize new views by manipulating specific latent variables.
• To enable zero-shot novel view generation from a single input image using a learned 3D rendering engine.
• To develop a training procedure that encourages neurons in the graphics code to represent specific, meaningful transformations without explicit supervision.
• To demonstrate that disentanglement is critical for accurate generalization to unseen object configurations and viewpoints.

Proposed method

• The model uses a deep encoder-decoder architecture with convolutional and de-convolutional layers, structured as a variational autoencoder with stochastic gradient variational Bayes (SGVB) for end-to-end training.
• The encoder maps input images to a posterior approximation over latent variables, while the decoder reconstructs images from these latents using de-convolutions and upsampling.
• A key training procedure activates only a subset of transformations (e.g., azimuth rotation) in each mini-batch, forcing specific neurons to represent only those active transformations.
• The objective function combines reconstruction loss and KL divergence between posterior and prior, enabling backpropagation through stochastic samples via reparameterization.
• Latent variables are grouped to represent distinct factors: e.g., one node for azimuth, others for shape, texture, and lighting.
• During inference, the encoder infers latents from a single image, and the decoder generates new images by modifying specific latent groups.

Experimental results

Research questions

• RQ1 Can a deep generative model learn a disentangled, interpretable graphics code from static images without explicit supervision?
• RQ2 Can such a model generalize to novel viewpoints and object configurations not seen during training?
• RQ3 Does enforcing disentanglement through selective transformation activation improve reconstruction and generation quality?
• RQ4 Can the model synthesize realistic novel views by manipulating only a subset of learned latents?
• RQ5 How well does the model perform on complex 3D objects like chairs when trained on a single image per object?

Key findings

• The DC-IGN achieved a test set mean-squared error (MSE) of 2.7722×10⁻⁴ on the 3D chairs dataset, indicating high-fidelity image reconstruction.
• On the face dataset, the model successfully generated novel views by manipulating only the azimuth-related latent, demonstrating equivariance and disentanglement.
• Explicit disentanglement was critical for accurate novel-view reconstruction, as shown by qualitative comparisons with non-disentangled baselines.
• The model could generalize to previously unseen chairs, generating plausible intermediate poses even when the object’s full geometry was not visible.
• For some chairs, the model produced smooth transitions across angles, while for others it generated only keyframe-like outputs, indicating limitations in generalizing to complex or ambiguous shapes.
• The model’s ability to infer hidden parts (e.g., arms on a chair) from a single view shows its capacity for 3D reasoning and plausible extrapolation.