[Paper Review] Fast and Accurate Optical Flow based Depth Map Estimation from Light Fields

Yang Chen, Martin Alain+1 more|arXiv (Cornell University)|Aug 11, 2020

Advanced Vision and Imaging27 references33 citations

TL;DR

This paper presents a depth estimation method for light fields by applying optical flow along an angular dimension, using CPM for initialization and feature-flow spatio-angular filtering to obtain dense and consistent disparity maps, which are converted to depth and can yield extra dense point clouds.

ABSTRACT

Depth map estimation is a crucial task in computer vision, and new approaches have recently emerged taking advantage of light fields, as this new imaging modality captures much more information about the angular direction of light rays compared to common approaches based on stereoscopic images or multi-view. In this paper, we propose a novel depth estimation method from light fields based on existing optical flow estimation methods. The optical flow estimator is applied on a sequence of images taken along an angular dimension of the light field, which produces several disparity map estimates. Considering both accuracy and efficiency, we choose the feature flow method as our optical flow estimator. Thanks to its spatio-temporal edge-aware filtering properties, the different disparity map estimates that we obtain are very consistent, which allows a fast and simple aggregation step to create a single disparity map, which can then converted into a depth map. Since the disparity map estimates are consistent, we can also create a depth map from each disparity estimate, and then aggregate the different depth maps in the 3D space to create a single dense depth map.

Motivation & Objective

• Motivate depth map estimation from 4D light fields leveraging their angular information.
• Develop a fast, accurate optical flow-based pipeline along the angular dimension of light fields.
• Combine coarse-to-fine patch matching with spatio-angular edge-aware filtering for dense, consistent disparity estimates.
• Aggregate multiple disparity maps to produce dense depth maps or dense 3D point clouds.

Proposed method

• Extract a 3D spatio-angular volume by fixing one angular dimension and taking a sequence of views.
• Use coarse-to-fine patch matching (CPM) to initialize optical flow between consecutive views.
• Apply an edge-aware spatio-angular filter (feature flow) to densify and regularize sparse matches along angular slices.
• Aggregate disparity estimates across angular views to form a final depth map via a simple median followed by a one-step variational refinement.
• Optionally fuse multiple depth maps to create an extra-dense 3D point cloud.

Experimental results

Research questions

• RQ1 Can angular-dimension optical flow provide accurate disparity estimates for light-field depth estimation?
• RQ2 Does initializing with CPM and filtering with feature flow yield a favorable speed/accuracy trade-off?
• RQ3 What is the impact of angular-consistency filtering on depth map quality and edge preservation?
• RQ4 Can multiple disparity estimates be fused to produce denser 3D point clouds without expensive global optimization?

Key findings

• The angular spatio-angular feature flow filtering significantly improves optical flow accuracy and consistency over initial sparse matches.
• CPM initialization offers best performance in the balance between speed and accuracy among tested initializations.
• A simple median aggregation plus one-step variational refinement suffices for a high-quality final depth map due to angular consistency.
• The method achieves competitive accuracy and speed compared with state-of-the-art light-field depth estimation methods on the HCI dataset.
• The approach enables extra-dense point clouds by aggregating multiple depth estimates in 3D space, outperforming single-depth-map density in some cases.
• Edge-preserving results are favorable on object boundaries and noisy scenes, though local filtering may occasionally be less smooth than global optimizations.