[Paper Review] VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation

Xiaoyu Shi, Zhaoyang Huang+8 more|arXiv (Cornell University)|Mar 15, 2023

Advanced Vision and Imaging4 citations

TL;DR

VideoFlow proposes a novel optical flow framework that concurrently estimates bi-directional optical flows for multiple consecutive video frames by leveraging temporal cues through a TRi-frame Optical Flow (TROF) module and a MOtion Propagation (MOP) module. It achieves state-of-the-art performance, reducing error by 15.1% and 7.6% on Sintel final and clean passes, and 19.2% on KITTI-2015, ranking first on all benchmarks.

ABSTRACT

We introduce VideoFlow, a novel optical flow estimation framework for videos. In contrast to previous methods that learn to estimate optical flow from two frames, VideoFlow concurrently estimates bi-directional optical flows for multiple frames that are available in videos by sufficiently exploiting temporal cues. We first propose a TRi-frame Optical Flow (TROF) module that estimates bi-directional optical flows for the center frame in a three-frame manner. The information of the frame triplet is iteratively fused onto the center frame. To extend TROF for handling more frames, we further propose a MOtion Propagation (MOP) module that bridges multiple TROFs and propagates motion features between adjacent TROFs. With the iterative flow estimation refinement, the information fused in individual TROFs can be propagated into the whole sequence via MOP. By effectively exploiting video information, VideoFlow presents extraordinary performance, ranking 1st on all public benchmarks. On the Sintel benchmark, VideoFlow achieves 1.649 and 0.991 average end-point-error (AEPE) on the final and clean passes, a 15.1% and 7.6% error reduction from the best-published results (1.943 and 1.073 from FlowFormer++). On the KITTI-2015 benchmark, VideoFlow achieves an F1-all error of 3.65%, a 19.2% error reduction from the best-published result (4.52% from FlowFormer++). Code is released at \url{https://github.com/XiaoyuShi97/VideoFlow}.

Motivation & Objective

• To address the limitation of existing optical flow methods that are restricted to two-frame estimation, ignoring valuable temporal cues from additional neighboring frames.
• To develop a unified framework that concurrently estimates bi-directional optical flows for multiple frames, enabling better motion modeling and robustness.
• To reduce ambiguity and improve flow accuracy in challenging regions such as occlusions and out-of-boundary pixels by leveraging wider temporal context.
• To design a scalable architecture that extends three-frame flow estimation to longer video sequences using dynamic motion propagation.

Proposed method

• The TRi-frame Optical Flow (TROF) module jointly estimates bi-directional optical flows from the center frame to its preceding and following frames, ensuring motion features are aligned at the same pixel across time.
• TROF uses a recurrent fusion mechanism to iteratively refine motion features by integrating bi-directional flow predictions and correlation features in a shared feature space.
• The MOtion Propagation (MOP) module connects multiple TROF units by propagating motion features along predicted flow trajectories using a motion state feature $\mathbf{M}_{t}^{k}$.
• MOP enables the temporal receptive field to grow over iterations, allowing information from distant frames to influence flow predictions in a differentiable and dynamic manner.
• The framework uses iterative refinement in a recurrent decoder, where motion features are warped between adjacent TROF units based on current flow estimates.
• The model is trained end-to-end with standard optical flow loss functions, and the bi-directional flow predictions are symmetric by design, enhancing consistency.

Experimental results

Research questions

• RQ1 Can jointly estimating bi-directional optical flows from a central frame improve motion consistency and reduce errors caused by misalignment in two-frame methods?
• RQ2 How can temporal context from multiple frames be effectively fused to improve flow estimation in occluded or ambiguous regions?
• RQ3 What is the optimal way to propagate motion features across multiple TROF units to extend three-frame flow estimation to longer video sequences?
• RQ4 Does maintaining a motion state feature during propagation lead to better performance than simple feature passing in multi-frame optical flow estimation?
• RQ5 Can a multi-frame optical flow framework outperform state-of-the-art two-frame models on standard benchmarks like Sintel and KITTI-2015?

Key findings

• VideoFlow achieves 1.649 average end-point-error (AEPE) on the Sintel final pass, a 15.1% reduction from the previous best result of 1.943.
• On the Sintel clean pass, VideoFlow achieves 0.991 AEPE, a 7.6% error reduction from the prior best result of 1.073.
• On the KITTI-2015 benchmark, VideoFlow achieves 3.65% F1-all error, a 19.2% reduction from the previous best result of 4.52%.
• The ablation study confirms that bi-directional flow estimation and recurrent fusion in TROF significantly improve performance, with the latter reducing AEPE by 0.07 on Sintel final pass.
• The MOP module with motion state feature propagation achieves the best performance, reducing F1-all error by 1.3% on KITTI-2015 compared to a naive feature-passing baseline.
• Bi-directional flow predictions are symmetric and highly accurate, with forward and backward flows achieving similar AEPE values, confirming the model's consistency and robustness.