[Paper Review] Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids

Qifeng Chen, Vladlen Koltun|arXiv (Cornell University)|Apr 12, 2016

Advanced Vision and Imaging3 references19 citations

TL;DR

This paper proposes Full Flow, a global optimization method for optical flow estimation that directly minimizes a classical Horn-Schunck-type objective over full regular grids without descriptor matching. By exploiting the regular structure of the grid, it achieves state-of-the-art performance on Sintel and KITTI 2015 benchmarks using only one-shot global optimization at a single resolution.

ABSTRACT

We present a global optimization approach to optical flow estimation. The approach optimizes a classical optical flow objective over the full space of mappings between discrete grids. No descriptor matching is used. The highly regular structure of the space of mappings enables optimizations that reduce the computational complexity of the algorithm's inner loop from quadratic to linear and support efficient matching of tens of thousands of nodes to tens of thousands of displacements. We show that one-shot global optimization of a classical Horn-Schunck-type objective over regular grids at a single resolution is sufficient to initialize continuous interpolation and achieve state-of-the-art performance on challenging modern benchmarks.

Motivation & Objective

• To address the limitation of local optimization in optical flow estimation, which often gets trapped in local minima.
• To eliminate the need for separate descriptor matching modules that are not optimized jointly with the flow objective.
• To demonstrate that global optimization over the full space of mappings on regular grids is computationally feasible and effective.
• To show that a minimalistic approach—optimizing the classical Horn-Schunck objective globally—yields state-of-the-art accuracy.
• To provide a simple, generalizable framework for optical flow that supports various data and regularization terms.

Proposed method

• Formulates optical flow estimation as a Markov Random Field (MRF) over a dense 2D grid of pixel correspondences.
• Optimizes the classical Horn-Schunck objective: $ E(f) = E_{\text{data}}(f) + \lambda E_{\text{reg}}(f) $, using global discrete optimization.
• Leverages the regular grid structure to accelerate message passing via min-convolution algorithms, reducing inner-loop complexity from quadratic to linear.
• Employs a patch-based, truncated NCC data term for improved robustness to large displacements and intensity variations.
• Uses the $ L^1 $-norm penalty for regularization to enhance robustness, especially in the presence of outliers.
• Applies continuous interpolation (e.g., Revaud et al.) to the globally optimized discrete flow for final high-precision output.

Experimental results

Research questions

• RQ1 Can global optimization of the classical Horn-Schunck objective over full regular grids achieve state-of-the-art optical flow accuracy?
• RQ2 Does eliminating descriptor matching and relying solely on global optimization of the flow objective yield better performance than current methods?
• RQ3 Can the regular structure of the grid be exploited to make global optimization computationally tractable for large-scale optical flow?
• RQ4 How do different data terms and regularization functions affect performance within a global optimization framework?
• RQ5 Is one-shot global optimization at a single resolution sufficient to achieve high accuracy without multi-scale refinement?

Key findings

• Global optimization of the classical Horn-Schunck objective over full regular grids achieves state-of-the-art performance on both the Sintel and KITTI 2015 benchmarks.
• The patch-based truncated NCC data term significantly outperforms the pixelwise Horn-Schunck data term, reducing average endpoint error (EPE) by over 10% compared to the baseline.
• The $ L^1 $-norm penalty for regularization yields the best performance, with the lowest average EPE of 5.847 on the Sintel benchmark.
• Replacing the pixelwise data term with a patch-based NCC term while retaining the classical quadratic regularizer achieves 90% of the top-performing variant’s accuracy.
• The method achieves high accuracy across all tested images, with the patch-based NCC term showing superior robustness on challenging scenes.
• The approach is highly efficient due to min-convolution acceleration, enabling global optimization over tens of thousands of nodes with linear complexity.