[Paper Review] E$^2$(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition

Chiara Plizzari, Mirco Planamente+5 more|arXiv (Cornell University)|Dec 7, 2021

Advanced Memory and Neural Computing4 citations

TL;DR

This paper introduces N-EPIC-Kitchens, the first event-based egocentric action recognition dataset, and proposes E²(GO)MOTION, a motion-augmented event stream framework that leverages event data for robust action recognition. By using event data alone or distilling optical flow via a teacher-student setup, the method achieves up to 4% higher accuracy than RGB-only models on unseen environments, outperforming even PWC-Net optical flow in real-time settings without flow computation at inference time.

ABSTRACT

Event cameras are novel bio-inspired sensors, which asynchronously capture pixel-level intensity changes in the form of "events". Due to their sensing mechanism, event cameras have little to no motion blur, a very high temporal resolution and require significantly less power and memory than traditional frame-based cameras. These characteristics make them a perfect fit to several real-world applications such as egocentric action recognition on wearable devices, where fast camera motion and limited power challenge traditional vision sensors. However, the ever-growing field of event-based vision has, to date, overlooked the potential of event cameras in such applications. In this paper, we show that event data is a very valuable modality for egocentric action recognition. To do so, we introduce N-EPIC-Kitchens, the first event-based camera extension of the large-scale EPIC-Kitchens dataset. In this context, we propose two strategies: (i) directly processing event-camera data with traditional video-processing architectures (E$^2$(GO)) and (ii) using event-data to distill optical flow information (E$^2$(GO)MO). On our proposed benchmark, we show that event data provides a comparable performance to RGB and optical flow, yet without any additional flow computation at deploy time, and an improved performance of up to 4% with respect to RGB only information.

Motivation & Objective

• To address the lack of event-based data in egocentric action recognition, especially under challenging conditions like fast motion and environmental shifts.
• To investigate whether event data can serve as a robust, low-latency alternative to optical flow in online egocentric action recognition.
• To develop methods that exploit the high temporal resolution and motion sensitivity of event cameras while avoiding expensive flow computation at test time.
• To evaluate the generalization of event-based models across seen and unseen environments, reducing environmental bias.
• To establish a benchmark for event-based vision in egocentric settings by releasing the N-EPIC-Kitchens dataset.

Proposed method

• The authors created N-EPIC-Kitchens by simulating event streams from RGB videos in the EPIC-Kitchens dataset, preserving multi-modal data including optical flow and audio.
• They propose E²(GO), a method that adapts 2D and 3D action recognition networks (I3D, TSM) to process event data directly by modifying convolutional layers to handle asynchronous event streams.
• They introduce E²(GO)MO, a distillation-based method that trains a student model on event data using optical flow (from TV-L1) as a teacher signal during training, enabling motion-aware representation without flow inference at test time.
• Event data is encoded into voxel grids using a temporal binning strategy to preserve motion dynamics while enabling compatibility with standard video models.
• The framework uses a teacher-student distillation pipeline where the student model learns to mimic the flow-aware representations of the teacher, using only offline flow computation during training.
• The method is evaluated on both seen and unseen environments to test robustness to domain shift, with ablation studies on modality fusion and model architecture.

Experimental results

Research questions

• RQ1 Can event data alone achieve competitive performance in egocentric action recognition compared to RGB and optical flow?
• RQ2 Does event-based learning generalize better than RGB-based models in unseen environments, reducing environmental bias?
• RQ3 Can distilling optical flow knowledge into event data during training enable high performance without requiring flow computation at inference time?
• RQ4 How does the performance of event-based models compare to end-to-end optical flow models (e.g., PWC-Net) in terms of accuracy and inference speed?
• RQ5 To what extent do event data and motion representations outperform appearance-based models in fast-motion, wearable scenarios?

Key findings

• E²(GO)MO achieves 53.98% accuracy on unseen environments using only event data and distilled flow knowledge, outperforming PWC-Net optical flow by up to 6% in accuracy while requiring no flow computation at test time.
• Event-only models (E²(GO)) achieve 38.76% accuracy on unseen environments, surpassing RGB-only I3D (35.65%) and TSM (35.97%) models, demonstrating robustness to domain shift.
• The E²(GO)MO method achieves 57.87% accuracy on seen data with only 6ms inference time per frame, compared to 488ms for TV-L1 optical flow, making it suitable for real-time applications.
• Event data alone achieves 55.54% accuracy on seen data, outperforming RGB-only I3D (58.49%) in some settings, indicating strong motion representation capability.
• Despite its high computational cost, TV-L1 optical flow still achieves the highest accuracy (58.47% on seen, 43.40% on unseen), suggesting that motion filtering in flow computation reduces noise and improves performance.
• The results demonstrate that event data is a viable, low-power alternative to optical flow, especially in online and resource-constrained egocentric systems.