[Paper Review] 3DCFS: Fast and Robust Joint 3D Semantic-Instance Segmentation via Coupled Feature Selection

Liang Du, Jingang Tan+6 more|arXiv (Cornell University)|Mar 1, 2020

3D Shape Modeling and Analysis47 references4 citations

TL;DR

This paper proposes 3DCFS, a fast and robust end-to-end framework for joint 3D semantic and instance segmentation using coupled feature selection. It introduces a novel CFSM module that adaptively fuses task-specific features via a gate mechanism, while a new loss function ($\mathcal{E}_{EMED}$) balances embedding dimension magnitudes to improve Euclidean distance reliability, achieving state-of-the-art accuracy, speed, and efficiency on S3DIS and ShapeNet datasets.

ABSTRACT

We propose a novel fast and robust 3D point clouds segmentation framework via coupled feature selection, named 3DCFS, that jointly performs semantic and instance segmentation. Inspired by the human scene perception process, we design a novel coupled feature selection module, named CFSM, that adaptively selects and fuses the reciprocal semantic and instance features from two tasks in a coupled manner. To further boost the performance of the instance segmentation task in our 3DCFS, we investigate a loss function that helps the model learn to balance the magnitudes of the output embedding dimensions during training, which makes calculating the Euclidean distance more reliable and enhances the generalizability of the model. Extensive experiments demonstrate that our 3DCFS outperforms state-of-the-art methods on benchmark datasets in terms of accuracy, speed and computational cost.

Motivation & Objective

• To address the limitations of sequential or naive joint learning in 3D semantic and instance segmentation, where upstream errors degrade downstream performance.
• To exploit reciprocal information between semantic and instance segmentation tasks by modeling their interdependence in a coupled, human-like perception-inspired manner.
• To improve the reliability of instance embedding similarity computation by balancing dimension magnitudes, which are otherwise prone to bias in Euclidean distance.
• To develop an efficient, end-to-end framework that achieves high performance with reduced computational cost and inference time.
• To demonstrate state-of-the-art performance across multiple benchmarks, including S3DIS and ShapeNet, in both accuracy and inference speed.

Proposed method

• Proposes a Coupled Feature Selection Module (CFSM) that uses two gate-controlled streams—one from instance to semantic and one from semantic to instance—to selectively fuse task-specific features.
• Employs a gate mechanism within CFSM to dynamically suppress irrelevant or conflicting features, enabling adaptive, task-aware feature fusion.
• Introduces a novel loss function $\mathcal{E}_{EMED}$ that encourages balanced magnitudes across embedding dimensions to stabilize and improve Euclidean distance-based clustering.
• Uses PointNet/PointNet++ as the backbone encoder and applies the CFSM in the decoder to jointly optimize both segmentation tasks end-to-end.
• Applies Euclidean distance on learned embeddings for instance clustering, with $\mathcal{E}_{EMED}$ ensuring that no single dimension dominates the distance computation.
• Employs a shared backbone and joint training to enable parameter efficiency and mutual supervision between tasks.

Experimental results

Research questions

• RQ1 Can a coupled, gate-based feature selection mechanism improve joint 3D semantic and instance segmentation beyond traditional multitask learning?
• RQ2 Does balancing the magnitude of embedding dimensions significantly enhance the reliability of instance clustering via Euclidean distance?
• RQ3 Can a human-inspired, reciprocal information flow between semantic and instance segmentation tasks lead to better performance and robustness?
• RQ4 How does the proposed 3DCFS framework compare in speed, accuracy, and memory efficiency to existing state-of-the-art methods on standard benchmarks?
• RQ5 To what extent does the $\mathcal{E}_{EMED}$ loss improve generalization and reduce sensitivity to embedding dimension scaling?

Key findings

• 3DCFS achieves 62.3 mAcc on S3DIS for semantic segmentation and 54.7 mPrec for instance segmentation, outperforming state-of-the-art methods in both metrics.
• The $\mathcal{E}_{EMED}$ loss successfully balances the mean magnitude of embedding dimensions, reducing variance and improving clustering reliability, as shown in Figure 6.
• On S3DIS, 3DCFS trains in only 26.4 minutes per epoch with 2,227 MB memory, significantly faster and more efficient than SGPN (59.3 min, 7,549 MB) and ASIS (64.7 min, 4,275 MB).
• Inference on S3DIS is also faster, consuming only 36.3 minutes and 307 MB, outperforming SGPN (209.5 min, 420 MB) and ASIS (54.2 min, 1,235 MB).
• On ShapeNet, 3DCFS achieves 87.6 mIoU for semantic segmentation, surpassing SGPN by 1.8 mIoU, demonstrating strong generalization on part-level segmentation.
• Qualitative results show that 3DCFS correctly segments complex objects like cars and chairs, avoiding over- or under-clustering errors common in baseline methods.