[Paper Review] Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos

Serena Yeung, Olga Russakovsky+4 more|arXiv (Cornell University)|Jul 21, 2015

Human Pose and Action Recognition39 references83 citations

TL;DR

This paper introduces MultiTHUMOS, a large-scale dataset of dense, multilabel action annotations in untrimmed internet videos, and proposes MultiLSTM, an LSTM-based model with extended temporal input and output connections to model complex temporal dependencies between actions. The method achieves improved action recognition accuracy and enables advanced tasks like action prediction, demonstrating significant gains over baseline models on dense, multilabel video understanding.

ABSTRACT

Every moment counts in action recognition. A comprehensive understanding of human activity in video requires labeling every frame according to the actions occurring, placing multiple labels densely over a video sequence. To study this problem we extend the existing THUMOS dataset and introduce MultiTHUMOS, a new dataset of dense labels over unconstrained internet videos. Modeling multiple, dense labels benefits from temporal relations within and across classes. We define a novel variant of long short-term memory (LSTM) deep networks for modeling these temporal relations via multiple input and output connections. We show that this model improves action labeling accuracy and further enables deeper understanding tasks ranging from structured retrieval to action prediction.

Motivation & Objective

• To address the limitations of existing action recognition datasets that focus on single, localized actions and lack dense, multilabel annotations.
• To enable comprehensive understanding of human activity by modeling multiple simultaneous and temporally connected actions in unconstrained videos.
• To develop a deep learning model capable of capturing complex temporal dependencies between actions across frames.
• To support advanced video understanding tasks such as structured retrieval and future action prediction.
• To provide a benchmark for dense, multilabel action detection in real-world, untrimmed video sequences.

Proposed method

• Extended the THUMOS dataset to create MultiTHUMOS, a new dataset with 65 action classes and 1.5 labels per frame, covering 30 hours of untrimmed video.
• Introduced MultiLSTM, a novel LSTM variant with multiple input and output connections to model temporal relations across and within action classes.
• Designed the model to handle dense, multilabel predictions by learning dependencies between actions in the same frame and in nearby frames.
• Enabled future and past action prediction by training the model with temporal offsets, using context windows from past or centered on the current frame.
• Utilized soft attention mechanisms to integrate temporal context from input and output sequences, improving modeling of action transitions.
• Evaluated performance using mean average precision (mAP) on dense action detection and action prediction tasks.

Experimental results

Research questions

• RQ1 Can a large-scale, dense, multilabel video dataset improve the performance of action recognition models in complex, real-world scenarios?
• RQ2 How effective is a modified LSTM architecture with extended input and output connections in modeling temporal dependencies between multiple actions in a single video frame?
• RQ3 To what extent can a model predict future or past actions based on learned temporal relationships in dense action sequences?
• RQ4 How do temporal modeling capabilities of MultiLSTM compare to strong baselines and prior knowledge (e.g., label distribution priors) in action prediction tasks?
• RQ5 What insights can be gained about action transition patterns and hierarchical relationships from a dataset with fine-grained and hierarchical action classes?

Key findings

• MultiLSTM achieved higher mAP than a plain LSTM baseline on the dense, multilabel action detection benchmark, demonstrating improved accuracy.
• The model achieved mAP ≈ 30% for predicting actions 0.5 seconds in the past, and maintained mAP ≈ 20–25% for predictions up to 2 seconds into the future.
• MultiLSTM outperformed a baseline model using ground-truth label distributions, especially in the immediate past (0–1 second), indicating superior temporal modeling.
• The model successfully predicted action transitions such as Jump → Fall and Dribble → Shot with high accuracy, as shown in qualitative examples.
• The dataset MultiTHUMOS contains 10.5 distinct action categories on average per video, compared to only 1.1 in THUMOS, enabling richer analysis of action interactions.
• Over 45% of frames in MultiTHUMOS contain two or more labels, validating the need for models that can handle dense, multilabel temporal reasoning.