[Paper Review] AEGNN: Asynchronous Event-based Graph Neural Networks

Simon Schaefer, Daniel Gehrig+1 more|arXiv (Cornell University)|Mar 31, 2022

Advanced Memory and Neural Computing4 citations

TL;DR

AEGNN introduces a novel graph neural network framework that processes asynchronous events as evolving spatio-temporal graphs, updating only affected nodes per event to drastically reduce computation and latency. It achieves up to 11-fold lower FLOPs and 8-fold lower latency than standard GNNs while matching or exceeding state-of-the-art performance on object classification and detection tasks.

ABSTRACT

The best performing learning algorithms devised for event cameras work by first converting events into dense representations that are then processed using standard CNNs. However, these steps discard both the sparsity and high temporal resolution of events, leading to high computational burden and latency. For this reason, recent works have adopted Graph Neural Networks (GNNs), which process events as ``static" spatio-temporal graphs, which are inherently "sparse". We take this trend one step further by introducing Asynchronous, Event-based Graph Neural Networks (AEGNNs), a novel event-processing paradigm that generalizes standard GNNs to process events as ``evolving" spatio-temporal graphs. AEGNNs follow efficient update rules that restrict recomputation of network activations only to the nodes affected by each new event, thereby significantly reducing both computation and latency for event-by-event processing. AEGNNs are easily trained on synchronous inputs and can be converted to efficient, "asynchronous" networks at test time. We thoroughly validate our method on object classification and detection tasks, where we show an up to a 11-fold reduction in computational complexity (FLOPs), with similar or even better performance than state-of-the-art asynchronous methods. This reduction in computation directly translates to an 8-fold reduction in computational latency when compared to standard GNNs, which opens the door to low-latency event-based processing.

Motivation & Objective

• Address the high computational cost and latency of processing event camera data using standard CNNs and GNNs.
• Overcome the inefficiency of recomputing full network activations for each event by enabling localized, event-driven updates.
• Preserve high temporal resolution and sparsity of event data while enabling efficient inference for real-time applications.
• Enable training on batched events with backpropagation and deployment in asynchronous mode without performance loss.
• Generalize standard GNNs to handle the dynamic, evolving nature of event streams through efficient, incremental updates.

Proposed method

• Represent events as spatio-temporal graphs where nodes correspond to pixels and edges to spatial proximity, with features encoding intensity changes.
• Introduce an asynchronous update rule that recomputes only the activations of nodes affected by a new event, minimizing redundant computation.
• Use SplineConv layers to incorporate spatial geometry into graph convolutions, improving feature representation through pseudo-coordinates and B-spline basis functions.
• Train the network on batched event data using standard backpropagation, then convert it to an asynchronous inference mode at test time.
• Apply max pooling and skip connections to improve feature hierarchy and gradient flow in both recognition and detection architectures.
• Optimize computational complexity by leveraging the sparsity of events and avoiding full forward passes after each event.

Experimental results

Research questions

• RQ1 Can graph neural networks be generalized to process events as evolving spatio-temporal graphs rather than static graphs?
• RQ2 Can localized, event-driven updates in GNNs significantly reduce computational complexity and inference latency compared to full recomputation?
• RQ3 Does the proposed asynchronous update mechanism maintain or improve performance on event-based vision tasks compared to state-of-the-art methods?
• RQ4 Can a model trained on batched events be effectively deployed in an asynchronous, event-by-event inference mode without performance degradation?
• RQ5 To what extent can the sparsity and high temporal resolution of event data be preserved and exploited in deep learning architectures?

Key findings

• AEGNN reduces computational complexity by up to 11-fold in FLOPs compared to standard GNNs on object classification and detection tasks.
• The method achieves an 8-fold reduction in computational latency relative to standard GNNs, enabling real-time processing.
• Performance on object classification and detection tasks matches or exceeds that of state-of-the-art asynchronous methods despite significantly lower computation.
• The model maintains identical output when deployed asynchronously after batch training, ensuring consistency across training and inference modes.
• SplineConv layers with B-spline basis functions and pseudo-coordinates improve feature representation by encoding spatial relationships between neighboring events.
• The use of skip connections and max pooling enhances feature learning and gradient flow, contributing to robust performance on complex tasks.