[Paper Review] Dense Scene Flow from Stereo Disparity and Optical Flow

René Schuster, Oliver Wasenmüller+1 more|arXiv (Cornell University)|Aug 30, 2018

Advanced Vision and Imaging15 references4 citations

TL;DR

This paper proposes a real-time dense scene flow estimation method by combining stereo disparity and optical flow via warping and edge-aware interpolation. Using SceneFlowFields (SFF) for interpolation, it achieves state-of-the-art performance on the KITTI benchmark with 6.6% D1 error, outperforming prior recombination methods while maintaining a fast inference time of 36 seconds.

ABSTRACT

Scene flow describes 3D motion in a 3D scene. It can either be modeled as a single task, or it can be reconstructed from the auxiliary tasks of stereo depth and optical flow estimation. While the second method can achieve real-time performance by using real-time auxiliary methods, it will typically produce non-dense results. In this representation of a basic combination approach for scene flow estimation, we will tackle the problem of non-density by interpolation.

Motivation & Objective

• To address the challenge of non-dense scene flow outputs from traditional stereo and optical flow combination methods.
• To enable real-time dense scene flow estimation by leveraging fast, accurate auxiliary methods for disparity and optical flow.
• To improve reconstruction quality of dense scene flow through advanced interpolation techniques applied to warped disparity and optical flow.
• To demonstrate that combining high-quality auxiliary results with sophisticated interpolation yields superior scene flow performance.

Proposed method

• The method warps the disparity map from time $t+1$ to time $t$ using optical flow to estimate the intermediate disparity $d_t^{t+1}$, forming a sparse scene flow representation.
• It uses bilinear interpolation to warp sub-pixel disparity values, though this leads to missing values at occlusions or image boundaries.
• Edge-aware interpolation via SceneFlowFields (SFF) is applied to reconstruct dense 3D geometry and motion fields from the sparse combination.
• The SFF algorithm performs two-step interpolation: first estimating local planes for geometry, then estimating local affine transformations for 3D motion.
• Multiple interpolation strategies are evaluated, including joint interpolation of geometry and motion, and separate interpolation of disparity and motion components.
• The approach integrates SPS-stereo for disparity and FlowFields++ for optical flow, both known for high accuracy and speed.

Experimental results

Research questions

• RQ1 Can a combination of stereo disparity and optical flow produce dense scene flow with competitive accuracy and real-time performance?
• RQ2 How does edge-aware interpolation compare to standard interpolation methods in reconstructing dense scene flow from sparse estimates?
• RQ3 Does joint interpolation of geometry and motion components yield better results than separate interpolation of disparity and motion?
• RQ4 To what extent do improvements in auxiliary tasks (disparity and optical flow) enhance the final scene flow quality in the recombination framework?
• RQ5 How does the proposed method compare to end-to-end scene flow networks in terms of accuracy and inference speed?

Key findings

• The proposed method achieves a D1 error of 6.6% on the KITTI Scene Flow Benchmark, outperforming the original SGM+SF combination by 2.1 percentage points in D1.
• The joint interpolation strategy (interpolating both geometry and motion) produced the best overall scene flow estimate despite slightly lower accuracy on individual sub-tasks.
• The method runs in 36 seconds, significantly faster than most end-to-end methods, and outperforms several state-of-the-art approaches in terms of speed-accuracy trade-off.
• Using advanced auxiliary networks (SPS-stereo and FlowFields++) improved the baseline performance compared to earlier methods like SGM and FF+.
• The interpolation of warped disparity using local plane models (disp-plane) and affine motion models (disp-affine) showed strong results, with disp-plane achieving 21.2% SF error and disp-affine 20.4%.
• The method demonstrates that high-quality auxiliary results combined with sophisticated interpolation can rival or surpass dedicated end-to-end scene flow networks in accuracy and efficiency.