[Paper Review] EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow

Jérôme Revaud, Philippe Weinzaepfel+2 more|arXiv (Cornell University)|Jan 12, 2015

Advanced Vision and Imaging21 references59 citations

TL;DR

EpicFlow proposes a novel optical flow estimation method that uses edge-preserving interpolation of sparse matches via an edge-aware geodesic distance to handle large displacements, occlusions, and motion boundaries. By initializing a one-level variational energy minimization with the interpolated dense correspondences, it achieves state-of-the-art performance on MPI-Sintel and competitive results on Kitti and Middlebury, with a runtime of just 16.4 seconds per image pair on a single CPU core.

ABSTRACT

We propose a novel approach for optical flow estimation , targeted at large displacements with significant oc-clusions. It consists of two steps: i) dense matching by edge-preserving interpolation from a sparse set of matches; ii) variational energy minimization initialized with the dense matches. The sparse-to-dense interpolation relies on an appropriate choice of the distance, namely an edge-aware geodesic distance. This distance is tailored to handle occlusions and motion boundaries -- two common and difficult issues for optical flow computation. We also propose an approximation scheme for the geodesic distance to allow fast computation without loss of performance. Subsequent to the dense interpolation step, standard one-level variational energy minimization is carried out on the dense matches to obtain the final flow estimation. The proposed approach, called Edge-Preserving Interpolation of Correspondences (EpicFlow) is fast and robust to large displacements. It significantly outperforms the state of the art on MPI-Sintel and performs on par on Kitti and Middlebury.

Motivation & Objective

• To address the limitations of coarse-to-fine variational optical flow methods, which suffer from error propagation and local minima in the presence of large displacements and motion discontinuities.
• To improve dense optical flow estimation by leveraging sparse matches and contour cues, exploiting the fact that motion boundaries often align with image edges.
• To develop a fast, accurate, and robust interpolation scheme that preserves edges and handles occlusions without relying on multi-scale optimization.
• To demonstrate that a one-level variational minimization initialized with edge-preserving interpolated matches outperforms traditional coarse-to-fine approaches.
• To provide an efficient alternative to coarse-to-fine schemes by reducing computational overhead while maintaining or improving accuracy.

Proposed method

• The method begins with a sparse set of feature matches, computed using a robust matching algorithm such as DeepMatching, which is resilient to large displacements and occlusions.
• It introduces an edge-aware geodesic distance metric that prioritizes paths along image contours, ensuring that motion discontinuities are preserved during interpolation.
• The dense correspondence field is generated by interpolating sparse matches using local affine models, where the geodesic distance replaces standard Euclidean distance to enforce edge consistency.
• An efficient approximation scheme is proposed to compute the geodesic distance using only one affine model per match, significantly accelerating the interpolation step without sacrificing accuracy.
• The interpolated dense correspondence field is used to initialize a one-level variational energy minimization, which refines the flow field using standard optical flow optimization.
• Image edges are extracted using the SED method to guide the geodesic distance computation, enhancing boundary preservation.

Experimental results

Research questions

• RQ1 Can a sparse-to-dense interpolation of matches using an edge-aware distance metric improve optical flow estimation in the presence of large displacements and occlusions?
• RQ2 Does replacing the standard Euclidean distance with an edge-aware geodesic distance lead to better preservation of motion boundaries in the interpolated flow field?
• RQ3 Can a one-level variational minimization initialized with edge-preserving interpolated matches outperform multi-scale coarse-to-fine approaches in accuracy and robustness?
• RQ4 Is it possible to accelerate the geodesic distance computation via an approximation scheme without degrading performance?
• RQ5 How does the proposed method compare to state-of-the-art optical flow methods on benchmark datasets like MPI-Sintel, Kitti, and Middlebury?

Key findings

• EpicFlow achieves the best performance on the MPI-Sintel test set, with an average end-point error (AEE) of 1.135 on non-occluded areas and 3.727 on occluded areas, outperforming all prior methods listed in Table 4.
• On the Kitti dataset, EpicFlow achieves an AEE of 1.5 on non-occluded areas and 3.8 overall, outperforming TF+OFM and DeepFlow, and matching the performance of NLTGV-SC while being faster.
• The method runs in 16.4 seconds per image pair (1024×436) on a single CPU core, with 91% of the time spent on matching, indicating high efficiency despite the dense interpolation step.
• The approach achieves an AEE below 0.4 pixels on the Middlebury dataset, demonstrating strong performance even in the absence of large displacements.
• Failure cases are primarily due to missing matches on thin structures (e.g., dragon’s spear and horns) or poor contour detection, which cause flow to leak into the background.
• The approximation of the geodesic distance enables fast computation with no loss in performance, making the interpolation step efficient and practical for real-world use.