[Paper Review] Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation

Giorgos Bouritsas, Sergiy Bokhnyak+3 more|arXiv (Cornell University)|May 8, 2019

3D Shape Modeling and Analysis58 references147 citations

TL;DR

This paper introduces Neural 3D Morphable Models (Neural3DMM), a novel framework that uses spiral convolutional networks to learn hierarchical, topology-aware representations of 3D deformable shapes directly on meshes. By enforcing consistent local vertex orderings via spiral scanning, the method breaks permutation invariance in graph neural networks, enabling anisotropic, lightweight, and efficient mesh convolutions that outperform linear Morphable Models and prior GNNs in 3D shape reconstruction and generation across human faces and bodies.

ABSTRACT

Generative models for 3D geometric data arise in many important applications in 3D computer vision and graphics. In this paper, we focus on 3D deformable shapes that share a common topological structure, such as human faces and bodies. Morphable Models and their variants, despite their linear formulation, have been widely used for shape representation, while most of the recently proposed nonlinear approaches resort to intermediate representations, such as 3D voxel grids or 2D views. In this work, we introduce a novel graph convolutional operator, acting directly on the 3D mesh, that explicitly models the inductive bias of the fixed underlying graph. This is achieved by enforcing consistent local orderings of the vertices of the graph, through the spiral operator, thus breaking the permutation invariance property that is adopted by all the prior work on Graph Neural Networks. Our operator comes by construction with desirable properties (anisotropic, topology-aware, lightweight, easy-to-optimise), and by using it as a building block for traditional deep generative architectures, we demonstrate state-of-the-art results on a variety of 3D shape datasets compared to the linear Morphable Model and other graph convolutional operators.

Motivation & Objective

• To address the limitation of permutation-invariant graph neural networks in 3D shape learning by encoding fixed mesh topology explicitly.
• To develop a deep generative model for 3D deformable shapes that operates directly on meshes without intermediate representations like voxels or 2D projections.
• To improve upon linear 3D Morphable Models and existing graph convolutional operators by enabling nonlinear, hierarchical feature learning on fixed-topology meshes.
• To demonstrate state-of-the-art performance in 3D shape reconstruction and generation using end-to-end autoencoders and GANs.

Proposed method

• Proposes a novel graph convolutional operator called spiral convolution that enforces consistent local ordering of mesh vertices via spiral scanning, breaking permutation invariance.
• Uses spiral scanning to define a 1-1 mapping between neighboring vertices and learnable filter parameters, enabling anisotropic filtering on meshes.
• Constructs hierarchical intrinsic mesh autoencoders using spiral convolutions as building blocks, forming the Neural3DMM framework.
• Employs the same architecture as COMA for fair comparison, but enhances it with a larger parameter space to boost representational capacity.
• Trains the model using a variational autoencoder objective with a reparameterization trick for latent space sampling.
• Applies the framework to both autoencoding and GAN-based generation, enabling shape arithmetic and identity synthesis in the latent space.

Experimental results

Research questions

• RQ1 Can a graph convolutional operator that explicitly encodes mesh topology through local ordering improve 3D shape representation learning compared to permutation-invariant GNNs?
• RQ2 Does a hierarchical autoencoder based on spiral convolutions achieve better 3D mesh reconstruction than linear Morphable Models and prior GNN-based autoencoders?
• RQ3 Can spiral convolution-based models generate realistic, diverse 3D facial identities with fine geometric details, outperforming traditional 3DMMs?
• RQ4 To what extent does the learned latent space support meaningful geometric operations like interpolation, extrapolation, and analogies?

Key findings

• Neural3DMM achieves state-of-the-art reconstruction error across all tested datasets, including COMA (faces), Mein3D (faces), and DFAUST (bodies), outperforming PCA, 3DMM, COMA, and ChebNet-based autoencoders.
• For a latent dimension explaining 85% of variance, Neural3DMM reduces mean reconstruction error by up to 30% compared to COMA and 40% compared to PCA on the COMA dataset.
• On the Mein3D dataset with high vertex count (over 50k), PCA becomes computationally infeasible, while Neural3DMM maintains high performance with significantly fewer parameters.
• The model enables high-quality shape arithmetic: smooth interpolations between identities and expressions, and plausible extrapolations that exaggerate facial features.
• GAN-based generation using the spiral convolutional generator produces realistic 3D faces with fine geometric details, making synthetic samples hard to distinguish from real scans, unlike the smoother, less realistic outputs of 3DMM.
• Even with latent spaces explaining over 99% of variance, Neural3DMM outperforms PCA due to its more efficient and semantically meaningful parameterization.