[Paper Review] FlowNet3D++: Geometric Losses For Deep Scene Flow Estimation

Zirui Wang, Shuda Li+3 more|arXiv (Cornell University)|Dec 3, 2019

Advanced Vision and Imaging4 references5 citations

TL;DR

FlowNet3D++ improves deep scene flow estimation by integrating geometric losses—point-to-plane distance and cosine angular alignment—into the FlowNet3D framework, boosting accuracy from 57.85% to 63.43% on the KITTI dataset. It further introduces a dynamic 3D reconstruction benchmark, demonstrating up to 15.0% lower reconstruction error than FlowNet3D and 35.2% improvement over KillingFusion.

ABSTRACT

We present FlowNet3D++, a deep scene flow estimation network. Inspired by classical methods, FlowNet3D++ incorporates geometric constraints in the form of point-to-plane distance and angular alignment between individual vectors in the flow field, into FlowNet3D. We demonstrate that the addition of these geometric loss terms improves the previous state-of-art FlowNet3D accuracy from 57.85% to 63.43%. To further demonstrate the effectiveness of our geometric constraints, we propose a benchmark for flow estimation on the task of dynamic 3D reconstruction, thus providing a more holistic and practical measure of performance than the breakdown of individual metrics previously used to evaluate scene flow. This is made possible through the contribution of a novel pipeline to integrate point-based scene flow predictions into a global dense volume. FlowNet3D++ achieves up to a 15.0% reduction in reconstruction error over FlowNet3D, and up to a 35.2% improvement over KillingFusion alone. We will release our scene flow estimation code later.

Motivation & Objective

• To improve the accuracy of deep point cloud-based scene flow estimation beyond the standard L2 loss.
• To incorporate geometric priors from classical registration methods into deep learning for better flow estimation.
• To develop a holistic benchmark for scene flow evaluation using dynamic 3D reconstruction, moving beyond isolated metrics.
• To design a pipeline that integrates sparse scene flow predictions into a dense global volume for reconstruction.
• To validate the effectiveness of geometric constraints through ablation and comparative experiments on public datasets.

Proposed method

• Introduces a point-to-plane (pp) loss term inspired by ICP, penalizing the perpendicular distance between predicted flow vectors and the plane defined by neighboring points.
• Applies a cosine distance (cos) loss to enforce angular alignment between predicted and ground truth flow vectors, improving directional accuracy.
• Combines L2, pp, and cos losses during training to jointly optimize for magnitude and direction accuracy.
• Proposes a novel dynamic integration pipeline that fuses sparse scene flow predictions into a global dense volumetric reconstruction using a deformable volume update strategy.
• Uses a multi-stage training schedule with extended 200 epochs to evaluate convergence and stability of geometric loss combinations.
• Employs the KITTI, Snoopy, and Duck datasets for evaluation, with ground truth meshes enabling quantitative reconstruction error analysis.

Experimental results

Research questions

• RQ1 Can geometric constraints such as point-to-plane distance and angular alignment improve deep scene flow estimation beyond L2 loss?
• RQ2 How do geometric losses affect training convergence and stability in 3D scene flow networks?
• RQ3 Can dynamic 3D reconstruction serve as a more holistic and practical benchmark for scene flow estimation than isolated metrics?
• RQ4 Does integrating scene flow predictions into a dense volume yield measurable improvements in reconstruction quality?
• RQ5 What is the optimal combination of geometric losses for maximizing scene flow accuracy and reconstruction fidelity?

Key findings

• The combination of point-to-plane and cosine losses with L2 loss increases scene flow accuracy from 57.85% to 63.43% on the KITTI dataset.
• The geometric loss combination improves training convergence and stability, with performance still growing at 200 epochs, unlike models trained with only L2 or single geometric losses.
• In dynamic 3D reconstruction, FlowNet3D++ reduces mean reconstruction error by up to 15.0% compared to FlowNet3D.
• FlowNet3D++ achieves up to a 35.2% improvement in reconstruction error over KillingFusion alone on the Snoopy and Duck datasets.
• The ablation study confirms that the combination of pp and cos losses yields the best performance, especially when training is extended to 200 epochs.
• The proposed average angle error metric effectively captures directional deviation, supplementing EPE and revealing limitations of EPE alone in evaluating flow direction accuracy.