[Paper Review] Latent feature disentanglement for 3D meshes

Jake Levinson, Avneesh Sud+1 more|arXiv (Cornell University)|Jun 7, 2019

3D Shape Modeling and Analysis57 references4 citations

TL;DR

This paper proposes a supervised generative 3D mesh model based on a mesh-convolutional Variational Autoencoder that explicitly disentangles shape and pose factors in the latent space. By training on synthetic and real datasets with controlled variations, the model achieves superior performance in disentangled generation, pose and shape transfer, temporal synchronization, and pose-invariant matching, outperforming baseline models even without direct task-specific training.

ABSTRACT

Generative modeling of 3D shapes has become an important problem due to its relevance to many applications across Computer Vision, Graphics, and VR. In this paper we build upon recently introduced 3D mesh-convolutional Variational AutoEncoders which have shown great promise for learning rich representations of deformable 3D shapes. We introduce a supervised generative 3D mesh model that disentangles the latent shape representation into independent generative factors. Our extensive experimental analysis shows that learning an explicitly disentangled representation can both improve random shape generation as well as successfully address downstream tasks such as pose and shape transfer, shape-invariant temporal synchronization, and pose-invariant shape matching.

Motivation & Objective

• To develop a generative 3D mesh model that learns disentangled representations of shape and pose factors.
• To enable user-controlled shape manipulation through explicit disentanglement of generative factors in the latent space.
• To improve downstream tasks such as pose transfer, temporal synchronization, and shape matching using disentangled representations.
• To overcome limitations of unsupervised disentanglement by using explicit supervision on generative factors during training.
• To enhance shape generation realism and robustness through a distortion-sensitive loss and improved latent sampling.

Proposed method

• Trains a mesh-convolutional VAE on large-scale datasets with controlled variations in shape and pose, using synthetic and real 3D meshes.
• Employs explicit supervision during training to associate specific latent factors with observable shape and pose variations.
• Incorporates a distortion-sensitive loss term to improve geometric realism in generated meshes.
• Uses an alternative latent sampling strategy to mitigate overparameterization and improve latent space efficiency.
• Enables disentangled generation by combining disentangled latent attributes from different source meshes (e.g., shape from one, pose from another).
• Applies dynamic time warping (DTW) using L² distances between latent pose encodings for temporal synchronization of motion sequences.

Experimental results

Research questions

• RQ1 Can explicit supervision on generative factors lead to better disentanglement in 3D mesh generation compared to unsupervised methods?
• RQ2 Does a disentangled latent representation improve performance on downstream tasks like pose and shape transfer?
• RQ3 Can disentangled latent spaces enable effective temporal synchronization of unaligned motion sequences?
• RQ4 How does disentanglement affect pose-invariant shape matching and recognition?
• RQ5 To what extent does disentanglement improve generalization and robustness in shape generation?

Key findings

• The disentangled model outperformed a directly trained baseline in pose and shape transfer, even without task-specific fine-tuning.
• The model achieved high accuracy in shape recognition, correctly identifying 50% of meshes in top-1 and 66% in top-3 matches using only shape encodings.
• Latent space interpolation produced semantically meaningful, realistic pose transitions, avoiding the mesh distortions seen in direct vertex-space interpolation.
• The model successfully synchronized unaligned motion sequences on the Faust dataset using latent pose encoding distances with DTW.
• Despite some training modes producing entangled latent spaces, the primary model achieved consistent disentanglement for both training and inference.
• The disentangled model achieved better performance than a vanilla VAE with identical architecture and capacity, validating the benefit of disentangled representations.