[Paper Review] MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization

Jiahui Huang, He Wang+5 more|arXiv (Cornell University)|Jan 17, 2021

3D Shape Modeling and Analysis83 references4 citations

TL;DR

MultiBodySync proposes an end-to-end, differentiable deep declarative network that jointly performs multi-body motion segmentation and rigid registration across multiple unordered 3D scans by leveraging spectral synchronization to enforce correspondence and segmentation consistency. It achieves state-of-the-art performance on both synthetic and real-world datasets, including a new benchmark (DynLab), with strong generalization to unseen object categories.

ABSTRACT

We present MultiBodySync, a novel, end-to-end trainable multi-body motion segmentation and rigid registration framework for multiple input 3D point clouds. The two non-trivial challenges posed by this multi-scan multibody setting that we investigate are: (i) guaranteeing correspondence and segmentation consistency across multiple input point clouds capturing different spatial arrangements of bodies or body parts; and (ii) obtaining robust motion-based rigid body segmentation applicable to novel object categories. We propose an approach to address these issues that incorporates spectral synchronization into an iterative deep declarative network, so as to simultaneously recover consistent correspondences as well as motion segmentation. At the same time, by explicitly disentangling the correspondence and motion segmentation estimation modules, we achieve strong generalizability across different object categories. Our extensive evaluations demonstrate that our method is effective on various datasets ranging from rigid parts in articulated objects to individually moving objects in a 3D scene, be it single-view or full point clouds.

Motivation & Objective

• To address the challenge of consistent motion-based segmentation and rigid registration across multiple unordered 3D scans with varying spatial arrangements.
• To enable robust, generalizable motion segmentation for novel object categories without requiring category-specific priors.
• To simultaneously recover consistent point correspondences and motion segmentation labels across multiple scans using a unified differentiable framework.
• To overcome limitations of existing methods that rely on sequential data or assume known semantics, by handling non-temporally dense, intermittent scans.

Proposed method

• Employs a differentiable architecture based on deep declarative networks to jointly optimize correspondence and motion segmentation.
• Uses 3D scene flow estimation to initialize pairwise correspondences between scans.
• Introduces two synchronization modules: one for weighted permutation synchronization of correspondences and another for segmentation label synchronization.
• Applies spectral synchronization to enforce global consistency of correspondences and segmentations across all scans.
• Explicitly decouples geometry and motion estimation, enabling generalization to unseen object categories.
• Uses iterative refinement with multiple synchronization steps to improve accuracy and robustness.

Experimental results

Research questions

• RQ1 Can a deep learning framework jointly and consistently estimate motion segmentation and rigid registration across multiple unordered 3D scans?
• RQ2 How can correspondence and segmentation consistency be enforced across multiple scans with large pose variations and partial views?
• RQ3 Can a method generalize to novel object categories without fine-tuning, especially for articulated and solid rigid bodies?
• RQ4 What is the impact of spectral synchronization on improving robustness and accuracy in multi-scan motion estimation?
• RQ5 How does the method perform under non-sequential, non-dense scan acquisition, such as in real-world scenarios?

Key findings

• MultiBodySync outperforms state-of-the-art methods by a large margin on both multi-body motion segmentation and motion estimation across diverse datasets.
• On the SAPIEN dataset, the method achieves a mean intersection-over-union (mIoU) of 66.7% on the overall category, significantly exceeding prior methods such as PointNet++ (47.5%) and NPP (48.2%).
• On the DynLab dataset, the method achieves a 95th percentile EPE3D of 0.066, demonstrating robustness under challenging conditions with large pose variations.
• The ablation study confirms that spectral synchronization significantly improves performance, with the 'S, NW' variant (spectral sync with normalization) achieving the best results.
• The method generalizes effectively to unseen categories, showcasing the first cross-category generalization for multi-scan multi-body motion segmentation in both articulated and object-level regimes.
• Visual results demonstrate robust segmentation and registration even in complex scenes with multiple moving objects and partial scans, including real-world data from the Waymo Open and DynLab datasets.