[Paper Review] Meta-Learning 3D Shape Segmentation Functions

Yu Hao, Huang, Hao+2 more|arXiv (Cornell University)|Oct 8, 2021

3D Shape Modeling and Analysis40 references4 citations

TL;DR

This paper proposes Meta-3DSeg, a meta-learning framework that enables fast adaptation of 3D shape segmentation models with minimal or no labeled data by learning a prior over the 3D segmentation function space. It uses a variational encoder to generate task-specific initialization for a segmentation MLP, achieving state-of-the-art unsupervised part segmentation performance on ShapeNet despite training without ground-truth labels.

ABSTRACT

Learning robust 3D shape segmentation functions with deep neural networks has emerged as a powerful paradigm, offering promising performance in producing a consistent part segmentation of each 3D shape. Generalizing across 3D shape segmentation functions requires robust learning of priors over the respective function space and enables consistent part segmentation of shapes in presence of significant 3D structure variations. Existing generalization methods rely on extensive training of 3D shape segmentation functions on large-scale labeled datasets. In this paper, we proposed to formalize the learning of a 3D shape segmentation function space as a meta-learning problem, aiming to predict a 3D segmentation model that can be quickly adapted to new shapes with no or limited training data. More specifically, we define each task as unsupervised learning of shape-conditioned 3D segmentation function which takes as input points in 3D space and predicts the part-segment labels. The 3D segmentation function is trained by a self-supervised 3D shape reconstruction loss without the need for part labels. Also, we introduce an auxiliary deep neural network as a meta-learner which takes as input a 3D shape and predicts the prior over the respective 3D segmentation function space. We show in experiments that our meta-learning approach, denoted as Meta-3DSeg, leads to improvements on unsupervised 3D shape segmentation over the conventional designs of deep neural networks for 3D shape segmentation functions.

Motivation & Objective

• To address the challenge of generalizing 3D shape segmentation across diverse shapes with limited or no labeled data.
• To formalize 3D shape segmentation as a meta-learning problem, learning priors over the function space rather than data distribution.
• To enable rapid adaptation of segmentation models to new shapes using only a few gradient steps.
• To improve unsupervised 3D part segmentation by learning a distribution over segmentation functions via a variational encoder.
• To achieve consistent and accurate part segmentation across shapes with significant structural variations.

Proposed method

• Formalizes 3D shape segmentation as a meta-learning problem, where each task is learning a unique segmentation function for a single 3D shape.
• Employs a self-supervised 3D shape reconstruction loss to train the 3D segmentation learner without requiring part-level labels.
• Introduces a 3D segmentation meta-learner composed of a variational autoencoder (VAE) that maps shape representations to a distribution over 3D segmentation tasks.
• Uses the VAE to sample task-specific priors that guide the initialization of the segmentation MLP parameters.
• Trains the meta-learner to predict optimal network weights for the segmentation learner via few-shot gradient updates on new shapes.
• Combines a shape encoder with a conditional MLP to represent the shape-conditioned segmentation function.

Experimental results

Research questions

• RQ1 Can meta-learning be effectively applied to 3D shape segmentation to enable fast adaptation with minimal labeled data?
• RQ2 Can learning priors over the 3D segmentation function space improve generalization across diverse shapes with structural variations?
• RQ3 How does a meta-learner that predicts initialization parameters compare to standard training on large labeled datasets in unsupervised settings?
• RQ4 Can a self-supervised reconstruction loss effectively train a segmentation function without part-level annotations?
• RQ5 Does the use of a variational encoder to model task distribution lead to better generalization than standard initialization strategies?

Key findings

• Meta-3DSeg achieves state-of-the-art unsupervised 3D part segmentation performance on the ShapeNet dataset, outperforming fully supervised methods under low-shot settings.
• The model trained in a purely unsupervised manner across all categories still achieves performance close to PointNet trained on all labeled data, demonstrating strong generalization.
• The use of a VAE-based meta-learner leads to better performance than standard weight initialization, with weight-setting-C showing superior results over weight-setting-B.
• Qualitative results show that Meta-3DSeg produces more accurate and consistent part-specific label predictions across diverse shapes, including complex objects like airplanes.
• The model generalizes effectively across categories without per-category fine-tuning, unlike category-specific models such as BAE-NET.
• The ablation study confirms that meta-learned initialization significantly improves segmentation performance compared to standard training, especially in low-data regimes.