[Paper Review] Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera

Lujie Xia, Ziluo Ding+6 more|arXiv (Cornell University)|Jul 12, 2023

CCD and CMOS Imaging Sensors4 citations

TL;DR

This paper proposes USFlow, an unsupervised optical flow estimation method for spike camera data using a dynamic timing representation based on multi-layer dilated convolutions and layer attention to capture multi-temporal-scale features. It achieves 15% and 19% error reduction over SCFlow on PHM dataset at Δt=10 and Δt=20, respectively, and demonstrates strong generalization on real and synthetic extreme driving scenarios via the proposed SSES dataset.

ABSTRACT

Efficiently selecting an appropriate spike stream data length to extract precise information is the key to the spike vision tasks. To address this issue, we propose a dynamic timing representation for spike streams. Based on multi-layers architecture, it applies dilated convolutions on temporal dimension to extract features on multi-temporal scales with few parameters. And we design layer attention to dynamically fuse these features. Moreover, we propose an unsupervised learning method for optical flow estimation in a spike-based manner to break the dependence on labeled data. In addition, to verify the robustness, we also build a spike-based synthetic validation dataset for extreme scenarios in autonomous driving, denoted as SSES dataset. It consists of various corner cases. Experiments show that our method can predict optical flow from spike streams in different high-speed scenes, including real scenes. For instance, our method gets $15\%$ and $19\%$ error reduction from the best spike-based work, SCFlow, in $Δt=10$ and $Δt=20$ respectively which are the same settings as the previous works.

Motivation & Objective

• To address the challenge of unsupervised optical flow estimation in high-speed scenes using spike camera data, which lacks labeled ground truth.
• To design a dynamic timing representation that efficiently extracts salient temporal features from spike streams with minimal parameters.
• To develop a spike-based unsupervised loss that estimates light intensity from spike intervals to enforce illuminance consistency.
• To build a synthetic dataset (SSES) for extreme autonomous driving scenarios to evaluate robustness in high-speed, rare, and dangerous events.
• To demonstrate state-of-the-art performance on real-world data, benchmark datasets (PHM), and the proposed SSES dataset.

Proposed method

• Proposes Temporal Multi-dilated Representation (TMR) using multi-layer dilated convolutions along the temporal dimension to extract features at varying receptive fields, enabling multi-temporal-scale feature learning.
• Introduces a Layer Attention (LA) module to dynamically fuse features from different dilated layers, filtering redundant information and emphasizing salient temporal patterns.
• Designs a novel unsupervised loss that estimates light intensity from spike intervals using two strategies: multi-interval-based and multi-time-window-based, adapting to regions with different motion speeds.
• Utilizes the estimated light intensity to compute illuminance consistency loss, enabling end-to-end training without ground truth optical flow.
• Generates the SSES synthetic dataset using CARLA simulator, simulating high-speed extreme driving scenarios with 40K Hz spike frames and 500 fps RGB frames.
• Trains on SPIFT and evaluates on PHM and SSES, using PWC-Flow as the backbone network for optical flow estimation.

Experimental results

Research questions

• RQ1 Can a dynamic timing representation based on dilated convolutions effectively extract multi-scale temporal features from spike streams for optical flow estimation?
• RQ2 How can unsupervised learning be adapted to spike-based data to estimate optical flow without ground truth supervision?
• RQ3 Can a spike-based unsupervised loss that estimates light intensity from spike intervals improve optical flow estimation performance?
• RQ4 Does the proposed method generalize well to real-world high-speed scenes and extreme autonomous driving scenarios?
• RQ5 Can a synthetic dataset of extreme driving scenarios (SSES) effectively validate the robustness of spike-based optical flow models?

Key findings

• USFlow achieves 1.307 mean AEE on the PHM dataset at Δt=10 and 2.413 at Δt=20, outperforming prior state-of-the-art spike-based method SCFlow by 15% and 19% error reduction, respectively.
• Abllation studies confirm that the proposed unsupervised loss with dynamic light intensity estimation significantly improves performance over ablated variants, especially in handling regions with varying motion speeds.
• The Layer Attention module enhances training stability and convergence speed, despite marginal performance gains, and contributes to better feature selection.
• USFlow demonstrates strong generalization on real-world data, producing visually accurate optical flow predictions even in complex, high-speed scenes.
• On the proposed SSES dataset, USFlow achieves competitive performance, with a mean AEE of 3.122 at Δt=10 and 3.130 at Δt=20, indicating robustness in extreme driving scenarios.
• The SSES dataset successfully captures diverse, high-speed, and rare accident-prone scenarios, enabling evaluation of model robustness under extreme conditions.