[Paper Review] Towards Anytime Optical Flow Estimation with Event Cameras

Yaozu Ye, Hao Shi+6 more|arXiv (Cornell University)|Jul 11, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes EVA-Flow, a deep learning framework for anytime optical flow estimation using event cameras. By introducing a Unified Voxel Grid representation and a Spatiotemporal Motion Refinement (SMR) module, EVA-Flow enables time-dense, low-latency (5ms) optical flow estimation at up to 200Hz, even when trained on low-frame-rate (10Hz) ground truth, achieving state-of-the-art accuracy and strong generalization on MVSEC, DSEC, and the authors' EVA-FlowSet dataset.

ABSTRACT

Event cameras respond to changes in log-brightness at the millisecond level, making them ideal for optical flow estimation. However, existing datasets from event cameras provide only low frame rate ground truth for optical flow, limiting the research potential of event-driven optical flow. To address this challenge, we introduce a low-latency event representation, Unified Voxel Grid, and propose EVA-Flow, an EVent-based Anytime Flow estimation network to produce high-frame-rate event optical flow with only low-frame-rate optical flow ground truth for supervision. Furthermore, we propose the Rectified Flow Warp Loss (RFWL) for the unsupervised assessment of intermediate optical flow. A comprehensive variety of experiments on MVSEC, DESC, and our EVA-FlowSet demonstrates that EVA-Flow achieves competitive performance, super-low-latency (5ms), time-dense motion estimation (200Hz), and strong generalization. Our code will be available at https://github.com/Yaozhuwa/EVA-Flow.

Motivation & Objective

• To address the limitation of existing event-based optical flow methods that are constrained by low-frame-rate ground truth (e.g., 10Hz or 20Hz), restricting high-temporal-resolution motion estimation.
• To enable anytime optical flow estimation—producing dense motion fields at arbitrary time intervals—despite supervision from low-frequency datasets.
• To design a network architecture that leverages the high temporal resolution and low-latency characteristics of event cameras to achieve real-time, time-dense motion estimation.
• To introduce an unsupervised evaluation metric, Rectified Flow Warp Loss (RFWL), for validating intermediate optical flow predictions without ground truth.

Proposed method

• Proposes a Unified Voxel Grid representation that ensures consistent event encoding across all temporal bins, improving feature quality compared to standard Voxel Grids.
• Introduces a stacked Spatiotemporal Motion Refinement (SMR) module that predicts temporally dense optical flow by iteratively refining motion estimates across time bins.
• Employs time-dense feature warping within the SMR module to provide implicit supervision for intermediate flow predictions, enhancing accuracy through spatial-temporal consistency.
• Uses a bin-by-bin inference strategy where the network processes event streams in fixed-duration bins, enabling flexible output frame rates by adjusting the number of bins.
• Leverages the Rectified Flow Warp Loss (RFWL) as an unsupervised metric to evaluate intermediate flow predictions during training, especially useful when ground truth is unavailable.
• Trains the model on low-frame-rate datasets (e.g., DSEC at 10Hz) but enables inference at much higher frame rates (up to 200Hz) by increasing the number of prediction bins.

Experimental results

Research questions

• RQ1 Can event-based optical flow networks achieve time-dense motion estimation beyond the frame rate of the training data’s ground truth?
• RQ2 How can implicit supervision from time-dense feature warping improve optical flow accuracy in event-based networks?
• RQ3 What is the impact of the Unified Voxel Grid representation on optical flow estimation performance compared to standard Voxel Grids?
• RQ4 Can an unsupervised evaluation metric like RFWL effectively validate intermediate optical flow predictions in the absence of ground truth?
• RQ5 How does increasing the number of prediction bins affect inference latency, accuracy, and generalization in anytime optical flow estimation?

Key findings

• EVA-Flow achieves an endpoint error (EPE) of 0.877 at 200Hz on the DSEC dataset with 21 bins, demonstrating high accuracy at ultra-high frame rates.
• The model achieves a latency of just 5ms and an inference speed of 9.2ms on a single GPU, enabling real-time anytime optical flow estimation.
• The Unified Voxel Grid reduces EPE by 7.3% compared to standard Voxel Grids, due to consistent event representation across all bins.
• As the number of bins increases from 6 to 21, EPE decreases from 0.955 to 0.877, indicating improved accuracy with higher temporal resolution, before slightly degrading at 31 bins due to sparse event content.
• The Rectified Flow Warp Loss (RFWL) enables effective unsupervised evaluation of intermediate flow predictions, validating the model’s internal consistency and motion refinement.
• The model generalizes well across datasets, achieving competitive performance on MVSEC, DSEC, and the newly introduced EVA-FlowSet, with strong robustness to varying event rates and scene dynamics.