[Paper Review] Self-Supervised 3D Scene Flow Estimation Guided by Superpoints

Yaqi Shen, Le Hui+2 more|arXiv (Cornell University)|May 4, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes SPFlowNet, an end-to-end self-supervised 3D scene flow estimation framework that iteratively generates dynamic superpoints guided by predicted flow to improve motion estimation accuracy. By jointly optimizing flow-guided superpoint generation and superpoint-guided flow refinement using a differentiable clustering mechanism and GRU-based refinement, the method achieves state-of-the-art performance on FlyingChairs, KITTI, and ScanNet benchmarks, outperforming prior methods by up to 9.92% in AR on KITTI.

ABSTRACT

3D scene flow estimation aims to estimate point-wise motions between two consecutive frames of point clouds. Superpoints, i.e., points with similar geometric features, are usually employed to capture similar motions of local regions in 3D scenes for scene flow estimation. However, in existing methods, superpoints are generated with the offline clustering methods, which cannot characterize local regions with similar motions for complex 3D scenes well, leading to inaccurate scene flow estimation. To this end, we propose an iterative end-to-end superpoint based scene flow estimation framework, where the superpoints can be dynamically updated to guide the point-level flow prediction. Specifically, our framework consists of a flow guided superpoint generation module and a superpoint guided flow refinement module. In our superpoint generation module, we utilize the bidirectional flow information at the previous iteration to obtain the matching points of points and superpoint centers for soft point-to-superpoint association construction, in which the superpoints are generated for pairwise point clouds. With the generated superpoints, we first reconstruct the flow for each point by adaptively aggregating the superpoint-level flow, and then encode the consistency between the reconstructed flow of pairwise point clouds. Finally, we feed the consistency encoding along with the reconstructed flow into GRU to refine point-level flow. Extensive experiments on several different datasets show that our method can achieve promising performance.

Motivation & Objective

• To address the limitation of fixed, offline superpoints in existing 3D scene flow methods, which often group points with dissimilar motions due to rigid clustering.
• To improve scene flow estimation accuracy by learning dynamic, flow-adaptive superpoints that reflect actual motion patterns in complex 3D scenes.
• To integrate superpoint generation and flow refinement into a single end-to-end, self-supervised framework for joint optimization.
• To reduce error propagation from inaccurate superpoints by using flow-predicted associations to guide online superpoint updates.
• To demonstrate state-of-the-art performance on synthetic and real-world LiDAR datasets without requiring ground-truth flow supervision.

Proposed method

• The framework uses farthest point sampling (FPS) to initialize superpoint centers with coordinates, flow, and features.
• A flow-guided superpoint generation module constructs soft point-to-superpoint associations using bidirectional flow predictions from the previous iteration.
• Superpoint centers are updated by adaptively aggregating features, coordinates, and flow values of associated points through the learned association map.
• Point-level flow is reconstructed by aggregating superpoint-level flow using the same association map.
• A consistency encoding layer measures alignment between reconstructed flows of source and target point clouds to enforce motion coherence.
• A GRU-based refinement module fuses reconstructed flow and consistency encoding to refine the final point-wise flow prediction.

Experimental results

Research questions

• RQ1 Can online, flow-guided superpoint generation improve 3D scene flow estimation accuracy compared to fixed, offline clustering?
• RQ2 How does integrating dynamic superpoints into an end-to-end learning framework affect performance on real-world and synthetic datasets?
• RQ3 What is the impact of superpoint consistency encoding and consistency loss on flow refinement quality?
• RQ4 How do hyperparameters such as the number of superpoints (L), neighborhood size (K), and number of iterations (T) affect model performance?
• RQ5 Can a self-supervised framework with no ground-truth flow supervision achieve state-of-the-art results on KITTI and FlyingChairs?

Key findings

• SPFlowNet achieves a 79.65% accuracy in average recall (AR) on the KITTI-t dataset, outperforming prior methods by up to 9.92% in AR.
• The model achieves an end-to-end self-supervised EPE of 0.089 on KITTI-r, demonstrating strong generalization without ground-truth supervision.
• Ablation studies show that removing the superpoint generation or consistency encoding leads to a significant drop in performance, with AR decreasing by 6.8% and 6.3% respectively.
• The optimal configuration uses L=30 superpoints, K=2 nearest centers, and T=3 iterations, balancing accuracy and inference efficiency.
• The flow-guided superpoint generation module outperforms SPNet-based superpoint generation, proving that flow guidance is critical when no ground-truth superpoints are available.
• The consistency loss is essential, as removing it increases EPE by 0.008 and reduces AR by 2.4%, confirming its role in enforcing motion coherence.