[Paper Review] Pushing the Limits of Asynchronous Graph-based Object Detection with Event Cameras

Daniel Gehrig, Davide Scaramuzza|arXiv (Cornell University)|Nov 22, 2022

Advanced Memory and Neural Computing22 citations

TL;DR

The paper introduces a scalable, efficient asynchronous graph neural network for event-based object detection that significantly increases depth and capacity while reducing per-event computation, achieving state-of-the-art accuracy on Gen1 and N-Caltech101 with much lower MFLOPS.

ABSTRACT

State-of-the-art machine-learning methods for event cameras treat events as dense representations and process them with conventional deep neural networks. Thus, they fail to maintain the sparsity and asynchronous nature of event data, thereby imposing significant computation and latency constraints on downstream systems. A recent line of work tackles this issue by modeling events as spatiotemporally evolving graphs that can be efficiently and asynchronously processed using graph neural networks. These works showed impressive computation reductions, yet their accuracy is still limited by the small scale and shallow depth of their network, both of which are required to reduce computation. In this work, we break this glass ceiling by introducing several architecture choices which allow us to scale the depth and complexity of such models while maintaining low computation. On object detection tasks, our smallest model shows up to 3.7 times lower computation, while outperforming state-of-the-art asynchronous methods by 7.4 mAP. Even when scaling to larger model sizes, we are 13% more efficient than state-of-the-art while outperforming it by 11.5 mAP. As a result, our method runs 3.7 times faster than a dense graph neural network, taking only 8.4 ms per forward pass. This opens the door to efficient, and accurate object detection in edge-case scenarios.

Motivation & Objective

• Motivate and address the efficiency gap in asynchronous GNNs for event cameras by enabling deeper, higher-capacity models without sacrificing speed.
• Propose architectural and computational tricks (pruning, early temporal aggregation, LUT-Spline Convolutions, directed event graphs) to maintain low per-event cost while increasing accuracy.
• Develop and evaluate multiple detector sizes (nano, small, medium, large) to demonstrate scalability and efficiency across regimes.
• Compare against state-of-the-art dense and sparse asynchronous methods on Gen1 and N-Caltech101 datasets to establish performance and efficiency gains.

Proposed method

• Represent events as directed spatio-temporal graphs with up to 50k nodes.
• Use Look-up-Table Spline Convolutions (LUT-SCs) as the core message-passing operator.
• Incorporate early temporal aggregation via max pooling to enable rapid information fusion and enable LUT-SC deployment.
• Apply node update pruning guided by pooling, position rounding, and feature changes to skip unnecessary computations (up to 73%).
• Deploy directed event graphs (DEGs) at input to stabilize and boost performance with minimal cost.
• Design a YOLOX-inspired multi-scale detection head operating on graph outputs to produce bounding boxes and class scores.

Experimental results

Research questions

• RQ1 How can depth and capacity be scaled in asynchronous graph-based networks for event cameras without skyrocketing computation?
• RQ2 What architectural changes (e.g., pruning, early aggregation, LUT-SCs, DEGs) provide the best trade-off between accuracy and efficiency?
• RQ3 Can an asynchronous GNN-based detector compete with dense and recurrent state-of-the-art methods on standard event datasets?
• RQ4 What is the impact of model size (nano to large) on mAP and MFLOPS/ev on Gen1 and N-Caltech101?

Key findings

• Small models achieve up to 3.7x lower computation while surpassing async baselines by 7.4 mAP on Gen1.
• Medium models are 13% more efficient than the most efficient prior method while beating it by 11.5 mAP.
• Large models outperform all dense methods and other sparse methods with 32.1 mAP on Gen1 and 73.2 mAP on N-Caltech101.
• Asynchronous processing runs 3.7x faster than a dense GNN (8.4 ms per forward pass).
• Ablation shows pruning via max pooling and early aggregation reduce MFLOPS/ev to as low as 4.58 with negligible mAP loss; LUT-SC reduces computation by ~4.5x compared to naive spline conv implementation.
• Directed event graphs provide a modest mAP boost (1.8 mAP) with small computational cost.