[Paper Review] Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks

Zheng Xu, Yexin Liu+6 more|arXiv (Cornell University)|Feb 17, 2023

Advanced Memory and Neural ComputingEngineering39 citations

TL;DR

This survey provides a comprehensive taxonomy of deep learning methods for event-based vision, benchmarks key DL approaches across reconstruction and scene understanding, and discusses challenges and future directions with an open-source repo.

ABSTRACT

Event cameras are bio-inspired sensors that capture the per-pixel intensity changes asynchronously and produce event streams encoding the time, pixel position, and polarity (sign) of the intensity changes. Event cameras possess a myriad of advantages over canonical frame-based cameras, such as high temporal resolution, high dynamic range, low latency, etc. Being capable of capturing information in challenging visual conditions, event cameras have the potential to overcome the limitations of frame-based cameras in the computer vision and robotics community. In very recent years, deep learning (DL) has been brought to this emerging field and inspired active research endeavors in mining its potential. However, there is still a lack of taxonomies in DL techniques for event-based vision. We first scrutinize the typical event representations with quality enhancement methods as they play a pivotal role as inputs to the DL models. We then provide a comprehensive survey of existing DL-based methods by structurally grouping them into two major categories: 1) image/video reconstruction and restoration; 2) event-based scene understanding and 3D vision. We conduct benchmark experiments for the existing methods in some representative research directions, i.e., image reconstruction, deblurring, and object recognition, to identify some critical insights and problems. Finally, we have discussions regarding the challenges and provide new perspectives for inspiring more research studies.

Motivation & Objective

• Provide a comprehensive overview of event representations and quality enhancement for DL inputs.
• Categorize DL methods into image reconstruction/restoration and scene understanding/3D vision.
• Benchmark representative DL methods to identify performance insights and gaps.
• Discuss challenges and future directions to guide future research in event-based DL.

Proposed method

• Classify event representations into six categories (image-based, surface-based, learning-based, voxel-based, graph-based, spike-based) and analyze their task suitability.
• Review quality enhancement techniques for noisy, low-resolution event data (denoising and super-resolution).
• Survey DL-based image/video reconstruction and event-guided SR/VSR approaches and compare performance under MSE, SSIM, LPIPS, and latency metrics.
• Summarize DL pipelines for event-based scene understanding tasks (classification, detection, tracking, segmentation, depth).
• Provide an open-source taxonomy and keep it updated with links to code via a public repo.

Experimental results

Research questions

• RQ1 How should event data be represented or converted to DNN-friendly inputs?
• RQ2 What advantages does deep learning bring over optimization-based methods for learning from events?
• RQ3 Is very deep neural modeling necessary for effective event-based vision?
• RQ4 How can DL methods balance the low-latency, high-tempo characteristics of event cameras with model complexity?
• RQ5 Are convolutional operations essential for filtering events, or can alternative architectures be more suitable?

Key findings

• Provide a comprehensive taxonomy of event representations and quality enhancement methods for DL-based event vision.
• Summarize DL-based approaches for image/video reconstruction and event-guided SR/VSR, highlighting strengths and limitations.
• Summarize DL-based methods for scene understanding and 3D vision using event data.
• Benchmark experiments (e.g., object recognition, reconstruction) reveal practical insights and remaining challenges.
• Introduce an open-source repository with taxonomy and code links to support ongoing research.