[Paper Review] LiteFlowNet3: Resolving Correspondence Ambiguity for More Accurate Optical Flow Estimation

Tak-Wai Hui, Chen Change Loy|arXiv (Cornell University)|Jul 18, 2020

Advanced Vision and Imaging30 references4 citations

TL;DR

LiteFlowNet3 improves optical flow estimation by introducing two novel modules: cost volume modulation to correct outlier-filled cost volumes using adaptive affine transformations guided by a confidence map, and flow field deformation that replaces inaccurate flows with consistent nearby flows via a displacement field. The method achieves state-of-the-art accuracy on Sintel and KITTI benchmarks while maintaining a small model size (7.7× smaller than HD3) and fast inference (2.2× faster).

ABSTRACT

Deep learning approaches have achieved great success in addressing the problem of optical flow estimation. The keys to success lie in the use of cost volume and coarse-to-fine flow inference. However, the matching problem becomes ill-posed when partially occluded or homogeneous regions exist in images. This causes a cost volume to contain outliers and affects the flow decoding from it. Besides, the coarse-to-fine flow inference demands an accurate flow initialization. Ambiguous correspondence yields erroneous flow fields and affects the flow inferences in subsequent levels. In this paper, we introduce LiteFlowNet3, a deep network consisting of two specialized modules, to address the above challenges. (1) We ameliorate the issue of outliers in the cost volume by amending each cost vector through an adaptive modulation prior to the flow decoding. (2) We further improve the flow accuracy by exploring local flow consistency. To this end, each inaccurate optical flow is replaced with an accurate one from a nearby position through a novel warping of the flow field. LiteFlowNet3 not only achieves promising results on public benchmarks but also has a small model size and a fast runtime.

Motivation & Objective

• Address correspondence ambiguity in optical flow estimation caused by occlusions and homogeneous image regions.
• Reduce the impact of outliers in the cost volume that degrade flow decoding accuracy.
• Improve flow accuracy by leveraging local flow consistency to correct erroneous flows in coarse-to-fine inference.
• Minimize model size and inference time while maintaining high performance on public benchmarks.
• Develop a lightweight, efficient network that avoids artifacts common in larger models like HD3.

Proposed method

• Apply adaptive affine transformation to each cost vector in the cost volume using a confidence map to identify unreliable flow regions.
• Use a confidence map to guide the generation of modulation parameters, ensuring only high-uncertainty regions are corrected.
• Introduce a novel meta-warping mechanism that deforms the flow field by replacing each flow with a more accurate one from a nearby position with similar features.
• Compute a displacement field via auto-correlation cost volume to guide the meta-warping, using the same confidence map for localization of errors.
• Integrate both cost volume modulation and flow field deformation into a coarse-to-fine framework to iteratively refine flow predictions.
• Train and evaluate the model end-to-end on FlyingChairs, with fine-tuning on Sintel and KITTI for benchmark performance.

Experimental results

Research questions

• RQ1 How can cost volume outliers caused by ambiguous feature matching be effectively mitigated in deep optical flow networks?
• RQ2 To what extent can local flow consistency improve optical flow accuracy in ill-posed correspondence regions?
• RQ3 Can a confidence map enhance the precision of cost volume modulation and flow field deformation?
• RQ4 How does the proposed method compare in accuracy, model size, and inference speed to state-of-the-art methods like HD3 and PWC-Net?
• RQ5 Does the integration of both modules lead to synergistic improvements in flow estimation across diverse datasets?

Key findings

• LiteFlowNet3 achieves a 3.91% AEE on Sintel Final (training set), outperforming PWC-Net+ (4.02%) and IRR-PWC (4.09%) with a 7.7× smaller model size than HD3.
• On KITTI 2015, LiteFlowNet3 achieves a 10.40% AEE, surpassing HD3 (10.65%) and SelFlow (10.70%) in the foreground region, despite using only two-frame training.
• The ablation study confirms that combining cost volume modulation and flow field deformation (CMFD) yields the best performance, reducing AEE by 0.25% on KITTI 2015 compared to CM alone.
• The confidence map significantly improves both modules: disabling it increases AEE by 0.07 on Sintel Final and 0.07 on KITTI 2015, showing its role in error localization.
• Visual results demonstrate that CMFD effectively removes green artifacts on Sintel and fills in missing flow on KITTI, especially around complex structures like windshields and signs.
• LiteFlowNet3 runs 2.2× faster than IRR-PWC (180ms) and 7.7× smaller than HD3 (39.9M parameters), while maintaining superior accuracy on all benchmarks.