[Paper Review] Weakly Supervised Learning of Rigid 3D Scene Flow

Leonidas Guibas, Tolga Birdal+3 more|arXiv (Cornell University)|Jan 1, 2021

Advanced Vision and ImagingComputer Science88 references4 citations

TL;DR

This paper proposes a weakly supervised deep learning method for 3D scene flow estimation that models scenes as rigidly moving objects, using only binary foreground/background masks and ego-motion annotations instead of dense flow labels. By enforcing rigidity at the object level and introducing test-time optimization, the approach achieves state-of-the-art performance on multiple autonomous driving benchmarks, reducing end-point error by over 30 cm on KITTI compared to prior methods.

ABSTRACT

We propose a data-driven scene flow estimation algorithm exploiting the observation that many 3D scenes can be explained by a collection of agents moving as rigid bodies. At the core of our method lies a deep architecture able to reason at the extbf{object-level} by considering 3D scene flow in conjunction with other 3D tasks. This object level abstraction, enables us to relax the requirement for dense scene flow supervision with simpler binary background segmentation mask and ego-motion annotations. Our mild supervision requirements make our method well suited for recently released massive data collections for autonomous driving, which do not contain dense scene flow annotations. As output, our model provides low-level cues like pointwise flow and higher-level cues such as holistic scene understanding at the level of rigid objects. We further propose a test-time optimization refining the predicted rigid scene flow. We showcase the effectiveness and generalization capacity of our method on four different autonomous driving datasets. We release our source code and pre-trained models under \url{github.com/zgojcic/Rigid3DSceneFlow}.

Motivation & Objective

• To address the challenge of accurate 3D scene flow estimation with minimal supervision, reducing reliance on expensive dense flow annotations.
• To improve generalization and robustness in dynamic 3D scene understanding by modeling motion as rigid transformations of objects.
• To enable effective training and inference on large-scale real-world datasets lacking dense scene flow annotations.
• To provide interpretable, object-level scene representations that support both low-level flow and high-level scene understanding.

Proposed method

• The method decomposes the scene into foreground (rigidly moving objects) and background (static), modeling background flow as ego-motion and foreground flow as rigid transformations per object.
• A deep neural network predicts transformation parameters (rotation and translation) for each segmented rigid agent, which are used to compute per-point rigid scene flow.
• The network is trained with weak supervision using only binary instance masks and ego-motion data, which are often available from IMUs or simple segmentation.
• A test-time optimization refines the predicted rigid scene flow by jointly optimizing object-level transformations and point-wise flow alignment.
• The method employs an entropy-regularized Sinkhorn algorithm to compute soft correspondence maps for robust ego-motion estimation, with slack rows and columns to down-weight outliers.
• The architecture is flexible and can be adapted to multiple 3D tasks, including semantic segmentation and instance-level rigidity loss.

Experimental results

Research questions

• RQ1 Can 3D scene flow estimation be effectively learned with only binary foreground/background masks and ego-motion annotations, rather than dense flow supervision?
• RQ2 How does modeling motion as rigid transformations of objects improve flow accuracy and generalization compared to unconstrained per-point flow prediction?
• RQ3 To what extent does test-time optimization refine predicted scene flow and improve alignment between consecutive LiDAR frames?
• RQ4 How well does the method generalize to new datasets without fine-tuning, especially when instance annotations are noisy or absent?

Key findings

• The method reduces end-point error by over 30 cm on the KITTI dataset compared to the previous state of the art, demonstrating significant performance gains.
• Even without ground-truth instance masks during training or inference, the method achieves performance comparable to models using GT masks, validating the effectiveness of the unsupervised clustering strategy.
• The ablation study confirms that the Sinkhorn algorithm significantly improves ego-motion estimation, reducing RTE and RRE by over 40% compared to using raw affinity matrices.
• Fine-tuning on the Waymo Open dataset improves robustness, especially for objects near the sensor and scenes with high object counts, outperforming direct generalization.
• The method generalizes effectively to the Waymo Open dataset without additional fine-tuning, showing strong zero-shot transfer capability.
• Failure cases are primarily due to noisy GT annotations in lidarKITTI (e.g., mislabeled object boundaries) or rare/rarely seen objects in Waymo, rather than inherent model limitations.