[Paper Review] DART: Distribution Aware Retinal Transform for Event-based Cameras

Bharath Ramesh, Hong Yang+4 more|arXiv (Cornell University)|Oct 30, 2017

Advanced Memory and Neural Computing52 references22 citations

TL;DR

This paper proposes DART, a distribution-aware retinal transform that uses log-polar grids to encode spatio-temporal event data from neuromorphic cameras, enabling robust feature descriptors for object classification, tracking, detection, and feature matching. DART achieves state-of-the-art performance on multiple event-based datasets, including 99% accuracy on MNIST-DVS and real-time classification on a UAV.

ABSTRACT

We introduce a generic visual descriptor, termed as distribution aware retinal transform (DART), that encodes the structural context using log-polar grids for event cameras. The DART descriptor is applied to four different problems, namely object classification, tracking, detection and feature matching: (1) The DART features are directly employed as local descriptors in a bag-of-features classification framework and testing is carried out on four standard event-based object datasets (N-MNIST, MNIST-DVS, CIFAR10-DVS, NCaltech-101). (2) Extending the classification system, tracking is demonstrated using two key novelties: (i) For overcoming the low-sample problem for the one-shot learning of a binary classifier, statistical bootstrapping is leveraged with online learning; (ii) To achieve tracker robustness, the scale and rotation equivariance property of the DART descriptors is exploited for the one-shot learning. (3) To solve the long-term object tracking problem, an object detector is designed using the principle of cluster majority voting. The detection scheme is then combined with the tracker to result in a high intersection-over-union score with augmented ground truth annotations on the publicly available event camera dataset. (4) Finally, the event context encoded by DART greatly simplifies the feature correspondence problem, especially for spatio-temporal slices far apart in time, which has not been explicitly tackled in the event-based vision domain.

Motivation & Objective

• To develop a generic, robust visual descriptor for event-based cameras that captures structural context using log-polar grids.
• To address the challenge of object recognition and tracking under extreme motion, occlusion, and viewpoint variation in event-based vision.
• To enable real-time, one-shot learning for long-term object tracking using event stream data.
• To create a benchmark for event-based long-term object tracking by augmenting ground truth annotations on a public dataset.
• To simplify spatio-temporal feature correspondence across distant time slices in event-based vision.

Proposed method

• DART maps past events into a log-polar grid centered on the most recent event, using spatial binning to form a descriptor based on interpolated event counts per bin.
• The descriptor is updated in real-time on an event-by-event basis using a queue to preserve precise space-time information.
• Normalization of the DART descriptor ensures robustness to intensity changes and motion, capturing relative event distribution.
• For classification, DART features are used in a bag-of-words framework with multi-scale testing to improve scale invariance.
• For tracking, a one-shot learning framework leverages statistical bootstrapping and circular shifts of DART descriptors to enhance robustness to rotation and scale.
• A long-term object tracking system (eLOT) combines a local search tracker with a global detection module using cluster majority voting to reinitialize after object re-entry.

Experimental results

Research questions

• RQ1 Can a log-polar based descriptor achieve robust object classification under varying scale, rotation, and viewpoint in event-based data?
• RQ2 How can one-shot learning be made robust for event-based object tracking using limited training data?
• RQ3 Can a hybrid tracker-detector framework effectively handle long-term object tracking with object re-entry and occlusion?
• RQ4 To what extent can DART descriptors enable reliable feature matching across temporally distant and spatially varying event streams?
• RQ5 Can the proposed system achieve real-time performance on embedded platforms like UAVs using event-based data?

Key findings

• DART achieved 99% classification accuracy on the MNIST-DVS dataset, outperforming existing methods.
• On CIFAR10-DVS, DART achieved 65.43% accuracy, demonstrating strong performance on complex object recognition tasks.
• On NCaltech-101, DART achieved 70.33% accuracy, showing effectiveness on natural image-like event data.
• The eLOT system demonstrated robust long-term tracking across diverse camera motion profiles, including 6-DOF, with high IoU scores on augmented ground truth.
• The system achieved real-time performance on a UAV using an Intel Compute Stick with an Intel Core m5-6Y57 vPro processor.
• Feature matching using DART successfully matched features across large temporal gaps and under scale/rotation changes, even with occlusion or fast camera motion.