[Paper Review] SiamEvent: Event-based Object Tracking via Edge-aware Similarity Learning with Siamese Networks

Yujeong Chae, Lin Wang+1 more|arXiv (Cornell University)|Sep 28, 2021

Advanced Memory and Neural Computing49 references4 citations

TL;DR

SiamEvent proposes a Siamese network-based event-based object tracking framework that leverages edge-aware similarity learning to track targets in dynamic and static scenes, including those with high dynamic range (HDR) and motion blur. By correlating embedded event features at two timestamps to compute edge similarity, it achieves up to 15% performance gain over state-of-the-art baselines on real-world and synthetic datasets.

ABSTRACT

Event cameras are novel sensors that perceive the per-pixel intensity changes and output asynchronous event streams, showing lots of advantages over traditional cameras, such as high dynamic range (HDR) and no motion blur. It has been shown that events alone can be used for object tracking by motion compensation or prediction. However, existing methods assume that the target always moves and is the stand-alone object. Moreover, they fail to track the stopped non-independent moving objects on fixed scenes. In this paper, we propose a novel event-based object tracking framework, called SiamEvent, using Siamese networks via edge-aware similarity learning. Importantly, to find the part having the most similar edge structure of target, we propose to correlate the embedded events at two timestamps to compute the target edge similarity. The Siamese network enables tracking arbitrary target edge by finding the part with the highest similarity score. This extends the possibility of event-based object tracking applied not only for the independent stand-alone moving objects, but also for various settings of the camera and scenes. In addition, target edge initialization and edge detector are also proposed to prevent SiamEvent from the drifting problem. Lastly, we built an open dataset including various synthetic and real scenes to train and evaluate SiamEvent. Extensive experiments demonstrate that SiamEvent achieves up to 15% tracking performance enhancement than the baselines on the real-world scenes and more robust tracking performance in the challenging HDR and motion blur conditions.

Motivation & Objective

• Address the limitation of existing event-based trackers that fail when targets stop or move non-independently in fixed scenes.
• Overcome the drifting problem in tracking by introducing target edge initialization and an edge detector.
• Enable robust tracking of arbitrary target edges regardless of motion type, including non-independent motion with background.
• Develop a comprehensive open-source dataset with both synthetic and real-world event sequences for training and evaluation.
• Improve tracking performance in challenging conditions such as HDR and motion blur by leveraging edge structure from sparse event streams.

Proposed method

• Embed event streams from two timestamps into feature tensors using event representation methods like EST, preserving edge structure.
• Feed the embedded target and search event features into a shared Siamese convolutional neural network to extract deep features.
• Correlate the shared features via cross-correlation to compute a similarity map indicating the most similar region to the target.
• Use edge-aware similarity learning to prioritize regions with matching edge structures, enhancing robustness to motion and illumination changes.
• Introduce target edge initialization to generate a clear, distinct edge representation of the target at the start of tracking.
• Implement an edge detector to classify whether the tracked edge is distinctive and sufficient, preventing drift in static scenes.

Experimental results

Research questions

• RQ1 Can edge-aware similarity learning in a Siamese network framework improve tracking robustness in event-based object tracking?
• RQ2 How does the proposed method perform on non-independent moving objects and fixed scenes where prior methods fail?
• RQ3 To what extent does edge structure information from sparse events improve tracking in high dynamic range and motion blur conditions?
• RQ4 What impact does target edge initialization and edge detection have on reducing tracking drift?
• RQ5 How does the performance of SiamEvent compare to state-of-the-art image-based and event-based trackers on real-world and synthetic data?

Key findings

• SiamEvent achieves a 15% performance improvement over state-of-the-art image-based and event-based trackers on real-world scenes.
• On the synthetic event dataset, SiamEvent achieves a success score of 0.404, precision score of 0.454, and success rate of 0.478, outperforming baselines.
• The edge-aware similarity learning mechanism enables robust tracking of targets in static scenes with moving cameras, such as in the 'arrow' and 'gs25' sequences.
• Target edge initialization improves the success score by 0.015 and precision score by 0.020, demonstrating its critical role in reducing drift.
• The edge detector reduces performance drop in static scenes, improving success rate from 0.441 (without) to 0.478 (with) on the synthetic dataset.
• SiamEvent operates at 25fps on real events (including embedding) and 120fps when events are pre-embedded, showing real-time feasibility.