[Paper Review] Technical Report: Temporal Aggregate Representations

Fadime Şener, Dibyadip Chatterjee+1 more|arXiv (Cornell University)|Jun 6, 2021

Human Pose and Action Recognition12 references4 citations

TL;DR

This paper proposes a multi-granular temporal aggregation framework for long-range video understanding, modeling both recent and long-term video contexts through scalable snippet features and non-local attention mechanisms. It achieves state-of-the-art performance on action anticipation and recognition tasks, outperforming prior methods by up to 11.4% on the Breakfast dataset and achieving 66.93% top-5 accuracy on EPIC-KITCHENS-100 through late fusion of RGB, flow, object, and ROI features.

ABSTRACT

This technical report extends our work presented in [9] with more experiments. In [9], we tackle long-term video understanding, which requires reasoning from current and past or future observations and raises several fundamental questions. How should temporal or sequential relationships be modelled? What temporal extent of information and context needs to be processed? At what temporal scale should they be derived? [9] addresses these questions with a flexible multi-granular temporal aggregation framework. In this report, we conduct further experiments with this framework on different tasks and a new dataset, EPIC-KITCHENS-100.

Motivation & Objective

• To address the challenge of long-term video understanding, where reasoning over extended temporal contexts is required for accurate action anticipation and recognition.
• To investigate how different temporal granularities and scopes—recent vs. spanning—can be effectively modeled in video understanding tasks.
• To evaluate the effectiveness of a flexible, multi-scale temporal aggregation framework across diverse long-range video benchmarks, including EPIC-KITCHENS-100 and Breakfast.
• To demonstrate the superiority of the proposed framework over existing methods like Timeception and PIC in modeling complex, long-duration activities.

Proposed method

• The method uses two types of snippet features: 'recent' snippets covering a few seconds before the current frame and 'spanning' snippets capturing long-term context up to ten minutes.
• It computes multi-scale representations by max-pooling frame features within snippets across varying temporal granularities (e.g., K={2,3,5} for recent, {7,5,3} for spanning).
• A Temporal Aggregation Block (TAB) couples each recent snippet with all spanning snippets via non-local blocks to model long-range dependencies and cross-modality attention.
• Coupling Blocks (CBs) apply non-local operations to learn relationships between recent and spanning features, with outputs from multiple TABs chained for final prediction.
• The model uses late fusion of predictions from multiple modalities—RGB, optical flow, object detections, and ROI features—on EPIC-KITCHENS-100 to improve robustness.
• The framework is trained with Adam optimizer, dropout (0.3), and learning rate decay, using 512-D linear layers for all non-classification heads.

Experimental results

Research questions

• RQ1 How can temporal relationships across multiple scales be effectively modeled in long-range video understanding?
• RQ2 What is the optimal temporal extent and granularity of context needed for accurate action anticipation and recognition in long videos?
• RQ3 How does multi-granular temporal aggregation compare to existing methods like Timeception and PIC in modeling long-duration activities?
• RQ4 To what extent do multi-modal features (RGB, flow, object, ROI) improve performance on egocentric video benchmarks like EPIC-KITCHENS-100?
• RQ5 Can the proposed framework generalize across different video tasks, including anticipation and recognition, without task-specific retraining?

Key findings

• On the Breakfast dataset, the proposed method achieves 80.8% accuracy without fine-tuning, outperforming Timeception [5] by 11.4% and the I3D baseline by 16.5%.
• With fine-tuned I3D features, the method achieves 89.8% accuracy, surpassing PIC [6] by 3.1% and demonstrating strong performance on complex, long-duration activities.
• On EPIC-KITCHENS-100, the late-fusion of all modalities (RGB, flow, object, ROI) achieves 66.93% top-5 accuracy on the validation set, significantly improving over individual modality results.
• The RGB-only model performs best for verb and noun recognition, while fusion improves performance on tail classes and unseen participants, indicating robustness to domain shift.
• The model achieves 62.68% top-1 and 64.05% top-5 accuracy on the test set for recognition, with 24.99% top-5 recall for unseen participants, showing strong generalization.
• In anticipation, the fusion model achieves 30.57% top-5 recall on the test set, with 25.26% for unseen participants, indicating effective modeling of future action prediction.