[Paper Review] VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows

Xiao Wang, Jianing Li+7 more|arXiv (Cornell University)|Aug 11, 2021

Advanced Memory and Neural Computing73 references20 citations

TL;DR

This paper proposes VisEvent, a large-scale benchmark for visible-event video tracking, and introduces a cross-modality transformer (CMT) to fuse features from RGB and event cameras. By leveraging the complementary strengths of both sensors—visible cameras for texture and event cameras for motion robustness—the method achieves state-of-the-art performance on VisEvent, with a success rate of 0.430 and precision of 0.632.

ABSTRACT

Different from visible cameras which record intensity images frame by frame, the biologically inspired event camera produces a stream of asynchronous and sparse events with much lower latency. In practice, visible cameras can better perceive texture details and slow motion, while event cameras can be free from motion blurs and have a larger dynamic range which enables them to work well under fast motion and low illumination. Therefore, the two sensors can cooperate with each other to achieve more reliable object tracking. In this work, we propose a large-scale Visible-Event benchmark (termed VisEvent) due to the lack of a realistic and scaled dataset for this task. Our dataset consists of 820 video pairs captured under low illumination, high speed, and background clutter scenarios, and it is divided into a training and a testing subset, each of which contains 500 and 320 videos, respectively. Based on VisEvent, we transform the event flows into event images and construct more than 30 baseline methods by extending current single-modality trackers into dual-modality versions. More importantly, we further build a simple but effective tracking algorithm by proposing a cross-modality transformer, to achieve more effective feature fusion between visible and event data. Extensive experiments on the proposed VisEvent dataset, FE108, COESOT, and two simulated datasets (i.e., OTB-DVS and VOT-DVS), validated the effectiveness of our model. The dataset and source code have been released on: \url{https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark}.

Motivation & Objective

• Address the lack of realistic, large-scale datasets for visible-event video tracking under challenging conditions such as low illumination, high speed, and background clutter.
• Develop a dual-modality tracking framework that effectively fuses visible and event data to improve tracking robustness.
• Design a simple yet effective cross-modality transformer module for interactive feature learning and fusion between RGB and event modalities.
• Establish a comprehensive benchmark and baseline methods to enable systematic evaluation and advancement of neuromorphic visual tracking.
• Demonstrate the superiority of the proposed method over existing single-modality and dual-modality trackers across multiple datasets and scenarios.

Proposed method

• Propose VisEvent, a large-scale benchmark with 820 video pairs (500 for training, 320 for testing) captured under low-light, high-speed, and cluttered conditions using DVS and RGB cameras.
• Transform event streams into event images by stacking them over fixed time windows to enable integration with deep learning models.
• Extend 30+ single-modality trackers into dual-modality versions using concatenation, element-wise addition, and 1×1 convolution fusion strategies.
• Design a cross-modality transformer (CMT) module that enables bidirectional feature interaction and attention-based fusion between visible and event features.
• Use a cross-attention mechanism in CMT to dynamically weigh relevant features across modalities, enhancing representation learning for tracking.
• Evaluate the model on VisEvent, FE108, COESOT, and simulated datasets (OTB-DVS, VOT-DVS) to validate robustness and generalization.

Experimental results

Research questions

• RQ1 Can a large-scale, realistic benchmark for visible-event video tracking be effectively constructed to support systematic evaluation of dual-modality tracking methods?
• RQ2 How does the fusion of visible and event modality features impact tracking performance under low-light, high-speed, and cluttered conditions?
• RQ3 Which feature fusion strategy—concatenation, addition, or attention-based mechanisms—yields the best performance in dual-modality tracking?
• RQ4 Can a cross-modality transformer module effectively learn and exploit complementary information from visible and event data for improved tracking accuracy?
• RQ5 To what extent does the proposed method generalize across real-world and simulated datasets, including VisEvent and existing benchmarks?

Key findings

• The proposed VisEvent benchmark contains 820 video pairs collected under realistic low-light, high-speed, and cluttered scenarios, providing a comprehensive testbed for dual-modality tracking.
• Among various fusion strategies, feature concatenation achieved the best performance with a success rate of 0.426 and precision of 0.627 on the VisEvent test set.
• The cross-modality transformer (CMT) achieved a success rate of 0.430 and precision of 0.632 on VisEvent, outperforming all baseline methods and existing SOTA trackers.
• The CMT-based tracker demonstrated superior robustness in handling motion blur, low illumination, and fast motion, as validated by qualitative visualizations.
• Failure cases were observed in scenarios with fast-moving small objects (e.g., baseball) and overlapping event streams (e.g., moving stars), indicating limitations in current event image representation.
• The study confirms that event cameras significantly enhance tracking performance in challenging visual conditions, especially when combined with visible data via effective fusion.