[Paper Review] Masked Event Modeling: Self-Supervised Pretraining for Event Cameras

Sinja Klenk, David Bonello+3 more|arXiv (Cornell University)|Dec 20, 2022

Advanced Memory and Neural Computing4 citations

TL;DR

This paper proposes Masked Event Modeling (MEM), a self-supervised pretraining framework for event cameras that reconstructs masked event sequences using a vision transformer backbone. By pretraining on large-scale unlabeled event data, MEM achieves state-of-the-art accuracy on N-ImageNet, N-Cars, and N-Caltech101, outperforming even supervised RGB pretraining on real-world event data and demonstrating strong label efficiency and generalization for semantic segmentation.

ABSTRACT

Event cameras asynchronously capture brightness changes with low latency, high temporal resolution, and high dynamic range. However, annotation of event data is a costly and laborious process, which limits the use of deep learning methods for classification and other semantic tasks with the event modality. To reduce the dependency on labeled event data, we introduce Masked Event Modeling (MEM), a self-supervised framework for events. Our method pretrains a neural network on unlabeled events, which can originate from any event camera recording. Subsequently, the pretrained model is finetuned on a downstream task, leading to a consistent improvement of the task accuracy. For example, our method reaches state-of-the-art classification accuracy across three datasets, N-ImageNet, N-Cars, and N-Caltech101, increasing the top-1 accuracy of previous work by significant margins. When tested on real-world event data, MEM is even superior to supervised RGB-based pretraining. The models pretrained with MEM are also label-efficient and generalize well to the dense task of semantic image segmentation.

Motivation & Objective

• To reduce dependency on costly and labor-intensive annotation of event data for deep learning tasks.
• To develop a self-supervised pretraining framework that leverages the temporal and spatial structure of unlabeled event sequences.
• To improve performance on downstream event-based vision tasks such as image classification and semantic segmentation.
• To achieve label efficiency and strong generalization across diverse event datasets and real-world recordings.
• To outperform existing supervised and self-supervised methods, including RGB-based pretraining, on event camera benchmarks.

Proposed method

• The method employs a masked autoencoding objective, where random temporal and spatial segments of event sequences are masked and the model reconstructs them.
• A vision transformer (ViT) backbone processes the event data, which is represented as event tensors with timestamped polarity events.
• The model is pre-trained on large-scale, diverse, unlabeled event recordings from various event cameras and environments.
• During pretraining, the model learns to predict missing event segments by attending to surrounding context through multi-head self-attention.
• After pretraining, the model is fine-tuned on downstream tasks with minimal labeled data, enabling strong few-shot and zero-shot performance.
• The framework is trained end-to-end using a reconstruction loss on masked event sequences, enabling representation learning without human-annotated labels.

Experimental results

Research questions

• RQ1 Can self-supervised pretraining on unlabeled event data improve performance on downstream event-based vision tasks?
• RQ2 How does MEM compare to supervised RGB pretraining on real-world event data?
• RQ3 To what extent does MEM generalize across different event datasets and downstream tasks?
• RQ4 How label-efficient is the MEM framework in low-data regimes?
• RQ5 Can masked reconstruction of event sequences learn meaningful spatiotemporal representations without supervision?

Key findings

• MEM achieves state-of-the-art top-1 accuracy on N-ImageNet, N-Cars, and N-Caltech101, significantly outperforming prior methods.
• On real-world event data, MEM surpasses supervised RGB-based pretraining in classification accuracy, demonstrating superior domain generalization.
• The model shows strong label efficiency, achieving high performance with minimal labeled data in fine-tuning.
• MEM generalizes well to dense prediction tasks, achieving strong results on semantic image segmentation with minimal adaptation.
• Pretraining on diverse, unlabeled event recordings leads to robust representations that transfer effectively across datasets and tasks.