[Paper Review] Event-based Simultaneous Localization and Mapping: A Comprehensive Survey

Kunping Huang, Sen Zhang+2 more|arXiv (Cornell University)|Apr 19, 2023

Advanced Memory and Neural Computing18 citations

TL;DR

This survey reviews event-based vSLAM systems that leverage asynchronous event streams, categorizing methods into feature-based, direct, motion-compensation, and deep learning approaches, and discusses representations, performance, challenges, and future directions.

ABSTRACT

In recent decades, visual simultaneous localization and mapping (vSLAM) has gained significant interest in both academia and industry. It estimates camera motion and reconstructs the environment concurrently using visual sensors on a moving robot. However, conventional cameras are limited by hardware, including motion blur and low dynamic range, which can negatively impact performance in challenging scenarios like high-speed motion and high dynamic range illumination. Recent studies have demonstrated that event cameras, a new type of bio-inspired visual sensor, offer advantages such as high temporal resolution, dynamic range, low power consumption, and low latency. This paper presents a timely and comprehensive review of event-based vSLAM algorithms that exploit the benefits of asynchronous and irregular event streams for localization and mapping tasks. The review covers the working principle of event cameras and various event representations for preprocessing event data. It also categorizes event-based vSLAM methods into four main categories: feature-based, direct, motion-compensation, and deep learning methods, with detailed discussions and practical guidance for each approach. Furthermore, the paper evaluates the state-of-the-art methods on various benchmarks, highlighting current challenges and future opportunities in this emerging research area. A public repository will be maintained to keep track of the rapid developments in this field at {\url{https://github.com/kun150kun/ESLAM-survey}}.

Motivation & Objective

• Explain the working principles and advantages of event cameras for vSLAM in high-speed and HDR scenarios.
• Classify and analyze the main event-based vSLAM approaches by methodology and data representations.
• Summarize performance on benchmarks and identify open challenges and opportunities for future research.
• Provide guidance and a public repository to track developments in event-based SLAM.

Proposed method

• Describe the event camera working principle and the event generation model including contrast thresholding and noise considerations.
• Review common event representations (individual events, event packets, event frames, time surfaces, motion-compensated frames, voxel grids, reconstructed images).
• Systematically review four categories of event-based vSLAM methods: feature-based, direct, motion-compensation, and deep learning methods, detailing their data association and optimization strategies.
• Discuss feature extraction, tracking, and camera tracking/mapping pipelines for event data, including point-based and line-based features, and learning-based approaches.
• Summarize state-of-the-art performance on datasets and benchmarks, and discuss challenges and future directions.

Experimental results

Research questions

• RQ1 What are the main representations of event data used in vSLAM, and how do they affect performance?
• RQ2 How do feature-based, direct, motion-compensation, and deep learning methods establish data associations and estimate camera pose and map from event streams?
• RQ3 What are the current challenges in event-based vSLAM, and what future directions appear most promising?
• RQ4 How do event-based methods compare with frame-based SLAM in benchmark scenarios?
• RQ5 What datasets and benchmarks are used to evaluate state-of-the-art event-based vSLAM methods?

Key findings

• Event cameras provide high temporal resolution, wide dynamic range, low latency, and low power consumption, enabling vSLAM in challenging illumination and motion conditions.
• Event-based vSLAM methods are categorized into four main types: feature-based, direct, motion-compensation, and deep learning approaches.
• A variety of event representations (individual events, event packets, event frames, time surfaces, voxel grids) are used to balance latency and computation.
• Feature-based methods extract and track features (points and lines) to estimate camera pose and 3D landmarks, sometimes using learning-based detectors and descriptors.
• Direct methods align event data via photometric or edge-based representations, while motion-compensation methods warp events to reference frames using motion models and IMU data when available.
• Deep learning approaches process event data with CNNs or SNNs to predict motion, depth, or representations, often trained with synthetic data and fine-tuned on real data.