[Paper Review] Learning Local Neighboring Structure for Robust 3D Shape Representation

Zhongpai Gao, Junchi Yan+4 more|arXiv (Cornell University)|Apr 21, 2020

3D Shape Modeling and Analysis4 citations

TL;DR

This paper proposes LSA-Conv, a local structure-aware anisotropic convolutional operation for 3D mesh representation learning that learns adaptive weighting matrices to soft-permute each vertex's neighbors, enabling shared anisotropic filtering without predefined coordinate systems. The method achieves state-of-the-art performance in 3D shape reconstruction and generalizes effectively to shape correspondence and monocular 3D face reconstruction tasks.

ABSTRACT

Mesh is a powerful data structure for 3D shapes. Representation learning for 3D meshes is important in many computer vision and graphics applications. The recent success of convolutional neural networks (CNNs) for structured data (e.g., images) suggests the value of adapting insight from CNN for 3D shapes. However, 3D shape data are irregular since each node's neighbors are unordered. Various graph neural networks for 3D shapes have been developed with isotropic filters or predefined local coordinate systems to overcome the node inconsistency on graphs. However, isotropic filters or predefined local coordinate systems limit the representation power. In this paper, we propose a local structure-aware anisotropic convolutional operation (LSA-Conv) that learns adaptive weighting matrices for each node according to the local neighboring structure and performs shared anisotropic filters. In fact, the learnable weighting matrix is similar to the attention matrix in the random synthesizer -- a new Transformer model for natural language processing (NLP). Comprehensive experiments demonstrate that our model produces significant improvement in 3D shape reconstruction compared to state-of-the-art methods.

Motivation & Objective

• Address the challenge of irregular, unordered neighbor structures in 3D meshes that hinder effective convolutional feature learning.
• Overcome limitations of isotropic filters and predefined local coordinate systems in existing graph CNNs for 3D shapes.
• Enable high-capacity, adaptive feature extraction by learning node-specific neighbor permutations without explicit ordering or fixed templates.
• Improve 3D shape reconstruction accuracy by leveraging local geometric structure through a differentiable, learnable weighting mechanism.
• Demonstrate generalization to downstream tasks such as 3D shape correspondence and monocular 3D face reconstruction.

Proposed method

• Propose LSA-Conv, a novel convolutional layer that learns a node-specific, differentiable weighting matrix to reweight and soft-permute neighbors based on local geometric structure.
• The weighting matrix is trained end-to-end as part of a deep network, analogous to the attention mechanism in Transformers, allowing adaptive neighbor ordering.
• Apply shared anisotropic filters on the reweighted neighbor set to extract local features, preserving spatial invariance and enhancing representational power.
• Integrate LSA-Conv into a convolutional mesh autoencoder (LSA-3DMM) for 3D shape reconstruction, maintaining compatibility with fixed-topology meshes.
• Use a learnable, parameter-efficient mechanism that avoids manual spiral ordering or kernel point assignment, unlike SpiralNet or KPConv.
• Train the model end-to-end using standard optimization on standard 3D mesh datasets with fixed topology, such as COMA and DFAUST.

Experimental results

Research questions

• RQ1 Can a learnable, structure-aware reweighting mechanism improve 3D mesh representation beyond fixed or isotropic convolutional operations?
• RQ2 Does eliminating the need for predefined local coordinate systems or spiral ordering enhance feature learning on irregular 3D meshes?
• RQ3 To what extent does the adaptive neighbor permutation via learnable weights improve reconstruction accuracy compared to isotropic or fixed-order methods?
• RQ4 Can LSA-Conv generalize effectively to downstream 3D vision tasks such as shape correspondence and monocular 3D face reconstruction?
• RQ5 How does the neighbor size (K) affect the trade-off between reconstruction accuracy and model complexity in the LSA-3DMM framework?

Key findings

• LSA-3DMM achieves state-of-the-art 3D shape reconstruction performance on both the COMA (face) and DFAUST (body) datasets, outperforming existing methods including COMA, SpiralNet, and PCA.
• On the COMA dataset, the reconstruction error of LSA-3DMM is 0.212, significantly lower than SpiralNet (0.227), demonstrating the superiority of learnable over predefined neighbor ordering.
• For the DFAUST dataset, LSA-3DMM achieves a reconstruction error of 3.869 with K=5, which is better than PCA (9.977), COMA (5.238), and Spiral (5.258), even with a small neighbor size.
• The ablation study confirms that removing the weighting matrix increases reconstruction error, proving the necessity of the structure-aware reweighting mechanism.
• In 3D shape correspondence, LSA-Conv reduces the average error to 2.501 cm on FAUST-inter, a 15% improvement over Groueix et al. (2.878 cm).
• In monocular 3D face reconstruction, the LSA-Conv-based decoder produces qualitatively superior results on AFLW2000-3D, validating its effectiveness in real-world 3D generation tasks.