[Paper Review] Displacement-Invariant Matching Cost Learning for Accurate Optical Flow Estimation

Jianyuan Wang, Yiran Zhong+4 more|arXiv (Cornell University)|Oct 28, 2020

Advanced Vision and Imaging45 citations

TL;DR

The paper introduces displacement-invariant cost learning (DICL) and a displacement-aware projection (DAP) layer to learn matching costs for optical flow without building a 5D feature volume, achieving state-of-the-art results on Sintel and KITTI.

ABSTRACT

Learning matching costs has been shown to be critical to the success of the state-of-the-art deep stereo matching methods, in which 3D convolutions are applied on a 4D feature volume to learn a 3D cost volume. However, this mechanism has never been employed for the optical flow task. This is mainly due to the significantly increased search dimension in the case of optical flow computation, ie, a straightforward extension would require dense 4D convolutions in order to process a 5D feature volume, which is computationally prohibitive. This paper proposes a novel solution that is able to bypass the requirement of building a 5D feature volume while still allowing the network to learn suitable matching costs from data. Our key innovation is to decouple the connection between 2D displacements and learn the matching costs at each 2D displacement hypothesis independently, ie, displacement-invariant cost learning. Specifically, we apply the same 2D convolution-based matching net independently on each 2D displacement hypothesis to learn a 4D cost volume. Moreover, we propose a displacement-aware projection layer to scale the learned cost volume, which reconsiders the correlation between different displacement candidates and mitigates the multi-modal problem in the learned cost volume. The cost volume is then projected to optical flow estimation through a 2D soft-argmin layer. Extensive experiments show that our approach achieves state-of-the-art accuracy on various datasets, and outperforms all published optical flow methods on the Sintel benchmark.

Motivation & Objective

• Motivate learning data-driven matching costs for optical flow, inspired by advances in stereo matching.
• Address the computational challenge of 2D displacement search by avoiding 5D feature volumes and 4D convolutions.
• Propose a displacement-invariant cost learning module that applies a 2D matching net per displacement hypothesis to construct a 4D cost volume.
• Introduce a displacement-aware projection layer to reweight costs along the displacement dimension and mitigate multi-modal ambiguities.
• Demonstrate improved accuracy across standard optical flow benchmarks and provide extensive ablations to validate the approach.

Proposed method

• Compute reference-target feature concatenations F_u(p) = F^1(p) || F^2(p+u) for each displacement u in a small 2D window.
• Apply a shared 2D matching network G to F_u to produce a cost C_u(p) for each displacement independently (no 5D volume or 4D convolutions).
• Aggregate costs across displacements with a displacement-aware projection (1x1 convolution over the displacement dimension) to yield C'_u.
• Project the 4D cost volume to a 2D optical flow via a 2D soft-argmin over the displacement set.
• Utilize a coarse-to-fine pyramidal warping strategy and a context network to refine flow predictions.

Experimental results

Research questions

• RQ1 Can learned matching costs be effectively estimated for optical flow without constructing a 5D feature volume or using 4D convolutions?
• RQ2 Does applying the same 2D matching net independently to each displacement hypothesis preserve useful information while reducing computation and memory?
• RQ3 Can a displacement-aware projection layer mitigate multi-modal issues and improve sub-pixel flow accuracy?
• RQ4 Do displacement-invariant cost learning (DICL) and the displacement-aware projection (DAP) yield state-of-the-art results on Sintel and KITTI datasets?
• RQ5 What are the comparative benefits of DICL over traditional dot-product/cosine or MLP-based cost learners in optical flow?

Key findings

• DICL achieves higher accuracy than non-learned metrics (dot product, cosine similarity) and other learned-cost baselines on FlyingChair, Sintel, and KITTI benchmarks.
• DAP consistently improves EPE by around 0.1 pixel across datasets and helps mitigate multi-modal effects in the soft-argmin estimation.
• On Sintel final, the method reaches 3.44 EPE, outperforming VCN and SelFlow, and on Sintel clean, 2.12 EPE, indicating strong generalization.
• Replacing non-learned metrics in PWCNet and VCN with DICL yields notable performance gains (e.g., PWCNet+DICL and VCN+DICL show improved Chair EPE).
• The approach demonstrates state-of-the-art results on KITTI 2015 and Sintel, with competitive validation on training sets and robust generalization without dataset-specific tuning.