[Paper Review] Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis

Shuicheng Yan, Scott Reed+2 more|arXiv (Cornell University)|Jan 5, 2016

Face recognition and analysis30 references275 citations

TL;DR

Presents a recurrent convolutional encoder-decoder that synthesize rotated views of unseen 3D objects from a single image, with weakly supervised disentangling of identity and pose.

ABSTRACT

An important problem for both graphics and vision is to synthesize novel views of a 3D object from a single image. This is particularly challenging due to the partial observability inherent in projecting a 3D object onto the image space, and the ill-posedness of inferring object shape and pose. However, we can train a neural network to address the problem if we restrict our attention to specific object categories (in our case faces and chairs) for which we can gather ample training data. In this paper, we propose a novel recurrent convolutional encoder-decoder network that is trained end-to-end on the task of rendering rotated objects starting from a single image. The recurrent structure allows our model to capture long-term dependencies along a sequence of transformations. We demonstrate the quality of its predictions for human faces on the Multi-PIE dataset and for a dataset of 3D chair models, and also show its ability to disentangle latent factors of variation (e.g., identity and pose) without using full supervision.

Motivation & Objective

• Enable 3D view synthesis from a single image for object categories like faces and chairs.
• Learn long-term pose transformations via a recurrent architecture to traverse the pose manifold.
• Disentangle identity and pose factors without explicit supervision while predicting rotations.
• Demonstrate applicability to novel objects and cross-view recognition.
• Investigate interpolation over identity and pose to generate new object instances.

Proposed method

• Use a deep convolutional encoder to extract identity and pose features from an input image.
• Introduce pose units and action inputs to model local pose changes, forming a recurrent pose predictor.
• Apply a recurrent convolutional encoder-decoder with fixed identity units and transformed pose units to generate rotated views.
• Provide a control signal at each time step to move along the pose manifold.
• Train with curriculum learning by gradually increasing sequence length of rotation steps.
• Optimize pixel-wise reconstruction loss over all steps to encourage coherent long-term predictions.

Experimental results

Research questions

• RQ1 Can a recurrent convolutional encoder-decoder learn to rotate 3D objects and render new views from a single input image?
• RQ2 Does curriculum training help long-term pose trajectory prediction and disentangle identity from pose without explicit labels?
• RQ3 How well does the model perform on unseen object instances (faces, chairs) and across-view recognition?
• RQ4 Can the model interpolate identity and pose to generate novel objects while preserving realism?

Key findings

• The model produces high-quality, smooth renderings for unseen faces and chairs across multiple rotation steps.
• Curriculum training improves both image synthesis quality and pose-invariant recognition performance.
• Disentangled identity and pose representations enable cross-view recognition without class labels, with identity units providing strong view-invariant identity signals.
• Chair interpolation demonstrates smooth stylistic transitions while preserving rotated views.
• The approach achieves competitive cross-view recognition compared to a CNN and outperforms image-based baselines at larger viewpoint differences.