[Paper Review] Event-based Vision for Early Prediction of Manipulation Actions

Daniel Deniz, Cornelia Fermüller+3 more|arXiv (Cornell University)|Jul 26, 2023

Advanced Memory and Neural ComputingEngineering3 citations

TL;DR

This paper proposes an event-based vision system using a Transformer architecture to predict manipulation actions in real time from asynchronous neuromorphic sensor data. It achieves state-of-the-art performance by leveraging spatio-temporal dynamics in events, reducing prediction latency by 2–3 seconds compared to video-based models, and outperforming video-based Transformers by 7.6% on average in accuracy for fine-grained action recognition.

ABSTRACT

Neuromorphic visual sensors are artificial retinas that output sequences of asynchronous events when brightness changes occur in the scene. These sensors offer many advantages including very high temporal resolution, no motion blur and smart data compression ideal for real-time processing. In this study, we introduce an event-based dataset on fine-grained manipulation actions and perform an experimental study on the use of transformers for action prediction with events. There is enormous interest in the fields of cognitive robotics and human-robot interaction on understanding and predicting human actions as early as possible. Early prediction allows anticipating complex stages for planning, enabling effective and real-time interaction. Our Transformer network uses events to predict manipulation actions as they occur, using online inference. The model succeeds at predicting actions early on, building up confidence over time and achieving state-of-the-art classification. Moreover, the attention-based transformer architecture allows us to study the role of the spatio-temporal patterns selected by the model. Our experiments show that the Transformer network captures action dynamic features outperforming video-based approaches and succeeding with scenarios where the differences between actions lie in very subtle cues. Finally, we release the new event dataset, which is the first in the literature for manipulation action recognition. Code will be available at https://github.com/DaniDeniz/EventVisionTransformer.

Motivation & Objective

• To enable early and continuous prediction of human manipulation actions using event-based vision for real-time human-robot interaction.
• To address the challenge of low-latency action recognition in robotics, where delays hinder effective planning and collaboration.
• To develop a model that captures subtle dynamic cues in hand-object interactions, especially when actions differ only in motion patterns.
• To release the first public event-based dataset for fine-grained manipulation action recognition.
• To demonstrate the superiority of event-based processing over video-based approaches in capturing motion dynamics for action classification.

Proposed method

• The method employs a Vision Transformer (ViT) architecture that processes time surfaces generated from asynchronous event streams, enabling online, incremental inference.
• Events are aggregated into time surfaces over fixed intervals (e.g., 33 ms), which are then fed into the Transformer for spatio-temporal feature learning.
• The model performs online prediction, updating confidence scores with each new event, allowing early classification before action completion.
• Self-attention mechanisms in the Transformer identify salient spatio-temporal patterns, such as hand motion trajectories and object interaction phases.
• Ablation studies compare the event-based Transformer against a video-based Transformer trained on the same action dataset, using identical hardware and computational load.
• Attention maps and confidence evolution are analyzed to interpret model decisions and assess how the network focuses on dynamic cues like repetitive motions or initial hand poses.

Figure 1: Example of online prediction of a manipulation action (“play with a stone”) using a Transformer architecture from a stream of asynchronous events. Event-wise processing provides a prediction with every new event. Top: action frames and continuous stream of events from the neuromorphic sens

Experimental results

Research questions

• RQ1 Can event-based vision with a Transformer architecture achieve earlier and more accurate prediction of manipulation actions compared to video-based methods?
• RQ2 How do attention mechanisms in the Transformer model identify and prioritize spatio-temporal patterns relevant to action prediction?
• RQ3 To what extent do event-based sensors outperform frame-based cameras in recognizing subtle dynamic differences between manipulation actions on the same object?
• RQ4 What is the latency reduction achieved by online, event-driven inference compared to batch processing of video segments?
• RQ5 How does the model’s performance vary across cyclic versus discrete manipulation actions, and what attention patterns emerge for each?

Key findings

• The event-based Transformer achieved a 7.6% average accuracy gain over the video-based Transformer, demonstrating superior performance in recognizing fine-grained manipulation actions.
• The model reached 80% confidence in predicting the correct action label approximately 0.7 seconds after the hand-object contact, significantly earlier than video-based models.
• Online inference reduced prediction latency by 2–3 seconds compared to conventional classification, enabling real-time perception-action loops.
• Attention maps revealed that the model focuses on motion dynamics—such as the start and end of cyclic actions like shaking or poking—rather than static object appearance.
• For discrete actions, attention was drawn to the initial hand pose and the trajectory during manipulation, highlighting the model’s sensitivity to dynamic motion cues.
• Despite similar computational cost, the event-based model processed up to 4,750 time surfaces per second, operating faster than real-time with lower prediction latency.

Figure 2: Pipeline of the proposed Mobilenet-based Transformer architecture for online inference. The event camera triggers asynchronous events which are accumulated into a Time Surface with exponential decay [ 36 ] , in this example with a constant $\tau=33ms$ . A Mobilenet architecture extracts sp