[Paper Review] Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories

Tiange Luo, Kaichun Mo+5 more|arXiv (Cornell University)|Feb 16, 2020

3D Shape Modeling and AnalysisEngineering54 references18 citations

TL;DR

This paper proposes OpenSeg, a bottom-up 3D part discovery framework that iteratively merges sub-parts using a learnable merging policy, achieving state-of-the-art performance on unseen categories. By leveraging partness scores and structured merging with geometric and shape-aware criteria, OpenSeg generalizes effectively to novel categories without shape-level bias.

ABSTRACT

We address the problem of discovering 3D parts for objects in unseen categories. Being able to learn the geometry prior of parts and transfer this prior to unseen categories pose fundamental challenges on data-driven shape segmentation approaches. Formulated as a contextual bandit problem, we propose a learning-based agglomerative clustering framework which learns a grouping policy to progressively group small part proposals into bigger ones in a bottom-up fashion. At the core of our approach is to restrict the local context for extracting part-level features, which encourages the generalizability to unseen categories. On the large-scale fine-grained 3D part dataset, PartNet, we demonstrate that our method can transfer knowledge of parts learned from 3 training categories to 21 unseen testing categories without seeing any annotated samples. Quantitative comparisons against four shape segmentation baselines shows that our approach achieve the state-of-the-art performance.

Motivation & Objective

• Address the challenge of 3D part segmentation in open-vocabulary settings where categories are unseen during training.
• Overcome the limitations of existing methods that rely on shape-level inductive biases, which hinder generalization to novel categories.
• Develop a universal shape structure detector that segments shapes into meaningful parts using only part-level priors.
• Enable robust zero-shot part discovery by learning a merging policy that minimizes error propagation through iterative refinement.

Proposed method

• Initialize with sub-part proposals from a point cloud using a part proposal network.
• Use a partness score network (PointNet-based) to evaluate the quality of each sub-part, optimizing via ℓ₂ loss.
• Apply a MergeNet, a binary classifier, to decide whether to merge pairs of sub-parts based on geometric and structural compatibility.
• Implement a multi-stage merging strategy using thresholds for convexity and SDF (signed distance function) consistency to guide merging decisions.
• Iteratively merge sub-parts using a transition function T that replaces two sub-parts with their union, forming a Markov Decision Process (MDP).
• Train the merging policy via reinforcement learning with a sparse reward signal penalizing invalid merges (i.e., unions not in the set of valid sub-parts).

Experimental results

Research questions

• RQ1 Can a bottom-up merging framework generalize to 3D part discovery in categories not seen during training?
• RQ2 How can part-level priors be leveraged to reduce shape-level inductive bias and improve zero-shot generalization?
• RQ3 What is the impact of using a learnable merging policy with partness scores on segmentation accuracy and robustness?
• RQ4 How does iterative merging with confidence-based refinement reduce error propagation compared to single-stage merging?
• RQ5 To what extent can geometric and structural cues (e.g., convexity, SDF consistency) improve merging decisions in open-vocabulary settings?

Key findings

• OpenSeg achieves state-of-the-art performance on part segmentation for unseen categories, with an average mAP of 54.4% on the PartNet dataset at IoU threshold 0.5.
• The method outperforms prior SOTA methods like SGPN (O1: 54.4% vs. 50.0% mAP) across all three levels of granularity (coarse, middle, fine).
• The use of partness score regression improves merging reliability, especially in cases where sub-parts contain points from multiple instances.
• The multi-threshold merging strategy (θ₁=0.9, θ₂=0.8, θ₃=0.7) and SDF threshold (α=0.12) significantly enhance structural consistency and reduce fragmentation.
• The framework generalizes well to novel categories, demonstrating robustness even when training data does not include the test category.
• Ablation studies show that removing the partness score or using a simple threshold-based merge leads to performance degradation, validating the need for learned quality estimation.