[Paper Review] PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation

Fenggen Yu, Kun Liu+3 more|arXiv (Cornell University)|Mar 2, 2019

3D Shape Modeling and Analysis41 references4 citations

TL;DR

PartNet proposes a recursive neural network-based deep learning model for hierarchical, fine-grained 3D shape segmentation from point clouds, enabling flexible decomposition into an arbitrary number of parts based on structural complexity. It achieves state-of-the-art performance by using shared weights across a tree-structured hierarchy, propagating recursive context features and enhancing them with part-specific geometry, while jointly predicting decomposition types (adjacency/symmetry) and stopping criteria.

ABSTRACT

Deep learning approaches to 3D shape segmentation are typically formulated as a multi-class labeling problem. Existing models are trained for a fixed set of labels, which greatly limits their flexibility and adaptivity. We opt for top-down recursive decomposition and develop the first deep learning model for hierarchical segmentation of 3D shapes, based on recursive neural networks. Starting from a full shape represented as a point cloud, our model performs recursive binary decomposition, where the decomposition network at all nodes in the hierarchy share weights. At each node, a node classifier is trained to determine the type (adjacency or symmetry) and stopping criteria of its decomposition. The features extracted in higher level nodes are recursively propagated to lower level ones. Thus, the meaningful decompositions in higher levels provide strong contextual cues constraining the segmentations in lower levels. Meanwhile, to increase the segmentation accuracy at each node, we enhance the recursive contextual feature with the shape feature extracted for the corresponding part. Our method segments a 3D shape in point cloud into an unfixed number of parts, depending on the shape complexity, showing strong generality and flexibility. It achieves the state-of-the-art performance, both for fine-grained and semantic segmentation, on the public benchmark and a new benchmark of fine-grained segmentation proposed in this work. We also demonstrate its application for fine-grained part refinements in image-to-shape reconstruction.

Motivation & Objective

• To address the rigidity of existing deep learning models that are trained for fixed sets of semantic labels, limiting adaptability across different part counts.
• To exploit the hierarchical nature of shape decomposition by modeling it as a top-down recursive binary splitting process, improving segmentation accuracy through contextual constraints.
• To develop a flexible architecture that can segment shapes into an arbitrary number of parts based on structural complexity, rather than predefined labels.
• To enhance segmentation accuracy by integrating both global recursive context features and local part geometry at each decomposition node.
• To demonstrate the utility of fine-grained hierarchical segmentation in downstream tasks such as image-to-shape reconstruction with part-level refinement.

Proposed method

• The model uses a recursive neural network (RvNN) with shared weights across all nodes in the hierarchy to perform top-down binary decomposition of a 3D shape represented as a point cloud.
• At each node, a classifier predicts the decomposition type (adjacency or symmetry) and whether to stop (leaf node), enabling dynamic, structure-aware segmentation.
• Recursive context features are propagated from parent to child nodes, providing high-level structural cues that constrain lower-level segmentations.
• Local part shape features are extracted for each sub-part and fused with the recursive context features to improve localization accuracy at each node.
• The network is trained end-to-end with a multi-task loss combining node classification loss and binary point labeling loss at every node in the hierarchy.
• A hierarchical reconstruction loss is applied during refinement, using bottom-up composition of part point clouds to supervise the refinement of individual parts.

Experimental results

Research questions

• RQ1 Can a deep learning model perform flexible, hierarchical 3D shape segmentation without being constrained to a fixed set of semantic labels?
• RQ2 How can recursive context features from higher-level decompositions improve segmentation accuracy in lower-level parts?
• RQ3 Can joint prediction of decomposition type (adjacency/symmetry) and stopping criteria enhance fine-grained part segmentation?
• RQ4 To what extent does integrating local part geometry with global recursive context improve segmentation performance?
• RQ5 Can the hierarchical structure learned by PartNet be effectively used to refine low-quality 3D reconstructions from single-view images?

Key findings

• PartNet achieves state-of-the-art performance on both the public ShapeNet benchmark and a newly proposed fine-grained segmentation benchmark, demonstrating superior accuracy in both fine-grained and semantic segmentation.
• The model successfully segments complex shapes into an arbitrary number of parts depending on structural complexity, unlike fixed-label models that are limited to predefined part counts.
• The integration of recursive context features with local part geometry leads to more accurate and contextually consistent segmentations, especially at fine-grained levels.
• The model identifies symmetric and adjacent parts correctly, which improves labeling consistency and reduces errors in structurally symmetric shapes.
• In image-to-shape reconstruction, the method enables high-quality refinement of reconstructed point clouds by leveraging the hierarchical part decomposition for localized, structure-aware optimization.
• Ablation studies show that training with ground-truth hierarchies built via rule-based methods significantly improves performance, indicating the importance of structured supervision.