[Paper Review] Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision

Xinchen Yan, Shuicheng Yan+3 more|arXiv (Cornell University)|Dec 1, 2016

Advanced Vision and ImagingComputer Science316 citations

TL;DR

The paper introduces Perspective Transformer Nets (PTN) that reconstruct 3D volumes from a single image using a silhouette-based projection loss, enabling training without ground-truth 3D volumes and enabling multi-class generalization.

ABSTRACT

Understanding the 3D world is a fundamental problem in computer vision. However, learning a good representation of 3D objects is still an open problem due to the high dimensionality of the data and many factors of variation involved. In this work, we investigate the task of single-view 3D object reconstruction from a learning agent's perspective. We formulate the learning process as an interaction between 3D and 2D representations and propose an encoder-decoder network with a novel projection loss defined by the perspective transformation. More importantly, the projection loss enables the unsupervised learning using 2D observation without explicit 3D supervision. We demonstrate the ability of the model in generating 3D volume from a single 2D image with three sets of experiments: (1) learning from single-class objects; (2) learning from multi-class objects and (3) testing on novel object classes. Results show superior performance and better generalization ability for 3D object reconstruction when the projection loss is involved.

Motivation & Objective

• Motivate learning a 3D object representation from a single 2D image while disentangling intrinsic shape from extrinsic viewpoint and illumination.
• Propose an encoder-decoder architecture with a perspective transformation-based projection loss to supervise 3D reconstruction without explicit 3D ground truth.
• Demonstrate learning across single and multiple object categories and evaluate generalization to unseen categories.
• Show robustness of the approach when trained with partial views compared to full azimuth views.

Proposed method

• Use an encoder-decoder network that maps a 2D image to a 3D voxel volume.
• Introduce Perspective Transformer Networks that perform differentiable perspective projection of the predicted volume to 2D silhouettes.
• Define a projection loss as the mean squared error between projected silhouettes and ground-truth silhouettes across multiple views.
• Allow joint training with optional volumetric supervision via a combined loss that balances projection and volume losses.
• Pre-train the encoder to learn a view-invariant latent representation before training the volume decoder with perspective transformers.

Experimental results

Research questions

• RQ1 Can a 3D shape be reconstructed from a single 2D view without ground-truth 3D volumes using a silhouette-based projection loss?
• RQ2 Does a multi-class PTN generalize to unseen categories better than a volume-supervised or single-view approach?
• RQ3 How does training with projection loss alone compare to training with full 3D supervision across various object categories?
• RQ4 What is the impact of using partial views on single-view 3D reconstruction performance?
• RQ5 Does incorporating projection regularization improve robustness and generalization to novel viewpoints?

Key findings

• Models trained with projection loss alone (PTN-Proj) can reconstruct 3D shapes without 3D supervision; e.g., chair category results show PTN-Proj IU of 0.5712 (GT 310) and 0.5027 (GT 130).
• Combined loss (PTN-Comb) generally yields stronger performance than volume-only training in multiclass settings, e.g., chair results: 0.6435 (GT 310) and 0.5067 (GT 130).
• Volume-only CNN-Vol achieves 0.6390 (GT 310) and 0.4983 (GT 130) on chair in single-category experiments, illustrating comparable performance to projection-based methods in some cases.
• In multiclass experiments, PTN-Proj and PTN-Comb outperform or match CNN-Vol on several categories (e.g., airplane, bench, dresser, car, chair, display, lamp, loudspeaker, rifle, sofa, table, telephone, vessel) with reported IU scores across categories."
• Out-of-category/generalization: Training on multiple categories improves generalization to unseen categories compared to single-category training, and projection loss helps robustness on novel shapes (e.g., bed, bookshelf, cabinet, motorbike, train).