[Paper Review] PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clouds

Yi Wei, Ziyi Wang+3 more|arXiv (Cornell University)|Dec 2, 2020

Advanced Vision and Imaging49 references4 citations

TL;DR

PV-RAFT proposes a novel point-voxel correlation field mechanism to estimate 3D scene flow from point clouds by combining K-Nearest Neighbor (KNN)-based point correlations for local details and multi-scale voxelized correlation fields for long-range dependencies. The method achieves state-of-the-art performance on FlyingThings3D and KITTI Scene Flow 2015, outperforming prior methods by significant margins through iterative GRU-based refinement and a memory-efficient truncation strategy.

ABSTRACT

In this paper, we propose a Point-Voxel Recurrent All-Pairs Field Transforms (PV-RAFT) method to estimate scene flow from point clouds. Since point clouds are irregular and unordered, it is challenging to efficiently extract features from all-pairs fields in the 3D space, where all-pairs correlations play important roles in scene flow estimation. To tackle this problem, we present point-voxel correlation fields, which capture both local and long-range dependencies of point pairs. To capture point-based correlations, we adopt the K-Nearest Neighbors search that preserves fine-grained information in the local region. By voxelizing point clouds in a multi-scale manner, we construct pyramid correlation voxels to model long-range correspondences. Integrating these two types of correlations, our PV-RAFT makes use of all-pairs relations to handle both small and large displacements. We evaluate the proposed method on the FlyingThings3D and KITTI Scene Flow 2015 datasets. Experimental results show that PV-RAFT outperforms state-of-the-art methods by remarkable margins.

Motivation & Objective

• To address the challenge of estimating accurate 3D scene flow from irregular, unordered point clouds.
• To overcome the limitations of existing methods that rely only on local or coarse correlations by capturing both local and long-range all-pairs dependencies.
• To develop an efficient, memory-optimized framework that maintains high accuracy through iterative refinement and correlation truncation.
• To generalize well across diverse point cloud datasets, including synthetic (FlyingThings3D) and real-world (KITTI) scenarios.

Proposed method

• Introduces point-voxel correlation fields that integrate KNN-based point correlations for fine-grained local features and multi-scale voxelized correlation fields for long-range dependencies.
• Employs a GRU-based recurrent update mechanism to iteratively refine scene flow predictions using both point-based and voxel-based correlation features.
• Applies a truncation mechanism to retain only the top-M correlation scores, reducing memory usage without sacrificing performance.
• Uses a feature encoder to extract per-point features, followed by a refinement module to smooth the final scene flow output.
• Constructs a pyramid of voxel grids at multiple scales to model long-range correspondences across varying spatial resolutions.
• Leverages a coarse-to-fine iterative strategy with all-pairs field transforms inspired by RAFT, adapted for 3D point clouds.

Experimental results

Research questions

• RQ1 Can combining point-based and voxel-based correlation fields improve scene flow estimation in 3D point clouds compared to methods relying on only one modality?
• RQ2 How effective is the proposed point-voxel correlation field in capturing both local and long-range dependencies in irregular 3D point clouds?
• RQ3 To what extent does the truncation mechanism reduce memory consumption while preserving prediction accuracy in all-pairs correlation learning?
• RQ4 Does the iterative GRU-based refinement with joint point-voxel correlations lead to better generalization on diverse datasets like FlyingThings3D and KITTI?
• RQ5 Can the proposed method outperform existing state-of-the-art methods in terms of both accuracy and efficiency on standard benchmarks?

Key findings

• PV-RAFT achieves an average end-to-end error (EPE) of 0.0461 on the KITTI Scene Flow 2015 dataset, significantly outperforming prior methods such as FlowNet3D (0.0704) and PointPWC-Net (0.0614).
• On the FlyingThings3D dataset, PV-RAFT achieves an EPE of 0.0585 with a strict accuracy of 0.7113, demonstrating strong generalization across datasets.
• The ablation study confirms that both point-based and voxel-based correlation components are essential, as removing either leads to performance degradation.
• The truncation operation with M=512 achieves optimal balance—reducing memory usage to 10.7GB while maintaining the best performance (EPE: 0.0461, accuracy: 0.8169).
• Visualizations show that voxel-based correlations guide initial predictions when flows are inaccurate, while point-based correlations refine localization as predictions converge.
• The method generalizes well to real-world data, achieving state-of-the-art results on the KITTI benchmark, which features complex real-world motion and noise.