[Paper Review] Learning a Hierarchical Latent-Variable Model of 3D Shapes

Shikun Liu, C. Lee Giles+1 more|arXiv (Cornell University)|May 17, 2017

3D Shape Modeling and Analysis47 references3 citations

TL;DR

The paper proposes the Variational Shape Learner (VSL), a hierarchical variational autoencoder with skip-connections that learns disentangled, multi-level latent representations of 3D voxel shapes in an unsupervised manner. By enforcing a structured hierarchy in the latent space, VSL enables high-fidelity 3D generation, superior shape reconstruction from single images, and effective shape arithmetic, outperforming state-of-the-art models in unsupervised classification and 3D reconstruction with significantly fewer parameters.

ABSTRACT

We propose the Variational Shape Learner (VSL), a generative model that learns the underlying structure of voxelized 3D shapes in an unsupervised fashion. Through the use of skip-connections, our model can successfully learn and infer a latent, hierarchical representation of objects. Furthermore, realistic 3D objects can be easily generated by sampling the VSL's latent probabilistic manifold. We show that our generative model can be trained end-to-end from 2D images to perform single image 3D model retrieval. Experiments show, both quantitatively and qualitatively, the improved generalization of our proposed model over a range of tasks, performing better or comparable to various state-of-the-art alternatives.

Motivation & Objective

• To address the limitation of flat, single-level latent representations in existing 3D shape generative models, which restrict model expressiveness and generalization.
• To learn a hierarchical, disentangled latent representation of 3D shapes that captures multi-level abstractions, from edges to complex structures.
• To enable fully unsupervised 3D shape generation and reconstruction without relying on human-annotated keypoint, segmentation, or pose information.
• To improve performance in unsupervised 3D shape classification and single-image 3D reconstruction compared to state-of-the-art models.
• To demonstrate the utility of learned latent spaces through shape arithmetic and meaningful interpolations.

Proposed method

• The model uses a deep variational autoencoder architecture with skip-connections between encoder and decoder layers to enforce hierarchical abstraction in the latent space.
• Each layer of latent variables is constrained to model a distinct level of abstraction, with higher layers controlling lower-level features through conditional generation.
• The model is trained end-to-end using variational inference with amortized inference, minimizing a variational lower bound on the log-likelihood of the data.
• A hierarchical prior is implicitly enforced via architectural design, avoiding the need for complex structured priors or additional regularization.
• The model is jointly trained on 2D images and voxelized 3D shapes, enabling single-image 3D reconstruction without supervision on pose or keypoints.
• Latent space operations (e.g., vector arithmetic) are performed directly on the hierarchical latent codes to generate novel, semantically meaningful 3D shapes.

Experimental results

Research questions

• RQ1 Can a hierarchical latent-variable model learn more expressive and disentangled representations of 3D shapes than flat latent models?
• RQ2 Can skip-connections in a variational autoencoder effectively enforce a structured hierarchy in the latent space for 3D shape generation?
• RQ3 Does the proposed model achieve state-of-the-art performance in unsupervised 3D shape classification without using human-annotated supervision?
• RQ4 Can the model reconstruct 3D shapes from single 2D images with high fidelity, even under occlusion and varying lighting?
• RQ5 Do vector operations in the learned hierarchical latent space produce semantically meaningful and realistic 3D shape interpolations?

Key findings

• The VSL outperforms state-of-the-art unsupervised models in 3D shape classification, achieving a 92.4% accuracy on ModelNet40 with a latent code of less than 100 dimensions, compared to 2.5 million dimensions in 3D-GAN.
• In single-image 3D reconstruction on PASCAL 3D, the separately trained VSL surpassed the 3D-R2N2 ResNet model in 8 out of 10 categories, with a mean IoU of 0.619 compared to 0.571 for 3D-R2N2.
• The VSL achieved a 0.631 IoU for the 'aero' class and 0.657 for 'bike' in the separately trained setting, significantly outperforming the 3D-R2N2 LSTM variant (0.472 and 0.330, respectively).
• t-SNE visualization of the latent space shows clear class separation on both ModelNet10 and ModelNet40, indicating effective disentanglement and clustering of shape classes.
• Shape arithmetic experiments produced plausible, intuitive combinations of 3D shapes (e.g., 'car + airplane' yielding a flying vehicle-like object), demonstrating meaningful semantic structure in the latent space.
• The model generates realistic 3D shapes by sampling from the learned hierarchical latent manifold, with no need for matching to real shapes or post-processing.