[Paper Review] Re-Evaluating LiDAR Scene Flow for Autonomous Driving

Nathaniel Chodosh, Deva Ramanan+1 more|arXiv (Cornell University)|Apr 4, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper re-evaluates self-supervised LiDAR scene flow benchmarks, revealing that top-performing learning-based methods on synthetic datasets like stereoKITTI underperform on real-world data. It proposes a learning-free baseline using ICP for ego-motion estimation, ground removal, and test-time optimization, which outperforms all evaluated methods on real-world datasets including Waymo, Argoverse, and NuScenes.

ABSTRACT

Popular benchmarks for self-supervised LiDAR scene flow (stereoKITTI, and FlyingThings3D) have unrealistic rates of dynamic motion, unrealistic correspondences, and unrealistic sampling patterns. As a result, progress on these benchmarks is misleading and may cause researchers to focus on the wrong problems. We evaluate a suite of top methods on a suite of real-world datasets (Argoverse 2.0, Waymo, and NuScenes) and report several conclusions. First, we find that performance on stereoKITTI is negatively correlated with performance on real-world data. Second, we find that one of this task's key components -- removing the dominant ego-motion -- is better solved by classic ICP than any tested method. Finally, we show that despite the emphasis placed on learning, most performance gains are caused by pre- and post-processing steps: piecewise-rigid refinement and ground removal. We demonstrate this through a baseline method that combines these processing steps with a learning-free test-time flow optimization. This baseline outperforms every evaluated method.

Motivation & Objective

• To investigate the validity of widely used self-supervised LiDAR scene flow benchmarks, such as stereoKITTI and FlyingThings3D, in guiding real-world autonomous driving research.
• To identify flaws in current benchmark design—specifically, unrealistic dynamic motion patterns, artificial correspondences, and unrepresentative dynamic-to-static ratios—that distort performance evaluation.
• To evaluate state-of-the-art learning-based scene flow methods on real-world datasets (Argoverse 2.0, Waymo, NuScenes) to assess their real-world generalization capability.
• To demonstrate that pre- and post-processing steps—especially ego-motion estimation via ICP, ground removal, and piecewise-rigid refinement—contribute more to performance than learning-based components.
• To propose and validate a dataless baseline method that combines ICP, ground segmentation, and test-time optimization, achieving state-of-the-art performance without any training data.

Proposed method

• Uses Iterative Closest Point (ICP) for ego-motion estimation, showing it outperforms all learning-based ego-motion predictors on real-world data.
• Applies a learned ground segmentation model to identify and remove static ground points before flow estimation, reducing noise and improving accuracy.
• Implements a test-time optimization strategy that enforces piecewise-rigidity constraints on the estimated flow, refining results post-inference without learning.
• Combines ICP-based ego-motion correction, ground removal, and piecewise-rigid refinement into a single baseline pipeline that requires no training data.
• Employs coordinate networks for height-map estimation to improve ground segmentation and spatial consistency in the flow estimation process.
• Evaluates the baseline and 10+ state-of-the-art self-supervised methods on real-world datasets using dynamic point end-point error (EPE) as the primary metric.

Figure 1 : Recent self-supervised scene flow methods [ 60 , 30 , 26 , 17 ] typically focus on performance on stereoKITTI [ 35 ] . We call attention to several problematic aspects of its semi-synthetic construction. When evaluated on real-world datasets (Waymo, Argoverse, NuScenes) we find a negative

Experimental results

Research questions

• RQ1 Does high performance on synthetic benchmarks like stereoKITTI correlate with strong performance on real-world LiDAR scene flow datasets such as Waymo, Argoverse, and NuScenes?
• RQ2 Can a learning-free method that relies on classical optimization and pre/post-processing outperform state-of-the-art self-supervised learning-based scene flow models on real-world data?
• RQ3 To what extent do pre-processing (e.g., ground removal) and post-processing (e.g., piecewise-rigid refinement) contribute to performance gains compared to learning-based components?
• RQ4 Why do current benchmarks like stereoKITTI and FlyingThings3D fail to reflect the true challenges of LiDAR scene flow in autonomous driving scenarios?
• RQ5 Is the performance of modern learning-based scene flow models primarily due to architectural innovations or to effective use of classical geometric and optimization techniques?

Key findings

• Performance on stereoKITTI is negatively correlated with performance on real-world datasets: methods that perform best on stereoKITTI consistently underperform on Argoverse, Waymo, and NuScenes.
• ICP outperforms all tested learning-based ego-motion estimation methods on real-world data, including those explicitly designed for ego-motion prediction.
• The proposed learning-free baseline—combining ICP, ground removal, and test-time piecewise-rigid optimization—outperforms every evaluated self-supervised method on all real-world datasets.
• On the NuScenes dataset, the baseline achieves a dynamic point end-point error (EPE) of 0.158 m, surpassing the best self-reported result of 0.172 m from SLIM, a state-of-the-art self-supervised method.
• On the lidarKITTI benchmark, the baseline outperforms all but one of the self- and weakly-supervised methods, demonstrating superior generalization despite no training data.
• Ground segmentation achieves 99.3% accuracy on NuScenes and 99.4% on Waymo, with less than 1% of dynamic points misclassified as ground, indicating robustness to non-planar terrain.

Figure 2 : The correlation between performance on stereoKITTI and performance on real datasets as a function of how much those datasets violate the one-to-one correspondence assumption present in stereoKITTI. The more a dataset violates this assumption (higher chamfer distance due to sparser pointcl