[Paper Review] Spatio-Temporal Action Detection with Multi-Object Interaction

Huijuan Xu, Lizhi Yang+3 more|arXiv (Cornell University)|Apr 1, 2020

Human Pose and Action Recognition43 references6 citations

TL;DR

This paper proposes the STAR model, an end-to-end spatio-temporal action detection framework that performs simultaneous spatial and temporal regression to localize multi-object interactions in action tubes without relying on external object detectors. It achieves competitive performance on UCF101-24 (53.0% mAP@0.5) and introduces a new large-scale dataset, something-STAR, annotated with multi-object interaction tubes.

ABSTRACT

Spatio-temporal action detection in videos requires localizing the action both spatially and temporally in the form of an "action tube". Nowadays, most spatio-temporal action detection datasets (e.g. UCF101-24, AVA, DALY) are annotated with action tubes that contain a single person performing the action, thus the predominant action detection models simply employ a person detection and tracking pipeline for localization. However, when the action is defined as an interaction between multiple objects, such methods may fail since each bounding box in the action tube contains multiple objects instead of one person. In this paper, we study the spatio-temporal action detection problem with multi-object interaction. We introduce a new dataset that is annotated with action tubes containing multi-object interactions. Moreover, we propose an end-to-end spatio-temporal action detection model that performs both spatial and temporal regression simultaneously. Our spatial regression may enclose multiple objects participating in the action. During test time, we simply connect the regressed bounding boxes within the predicted temporal duration using a simple heuristic. We report the baseline results of our proposed model on this new dataset, and also show competitive results on the standard benchmark UCF101-24 using only RGB input.

Motivation & Objective

• To address the limitation of existing spatio-temporal action detection methods that focus on single-person actions and fail on multi-object interactions.
• To define actions not just as movements of a single subject, but as interactions between multiple objects, requiring broader spatial localization.
• To develop a model that regresses both spatial and temporal boundaries simultaneously, avoiding reliance on external detectors or complex post-processing.
• To create a new large-scale dataset, something-STAR, with action tubes annotated to include both subjects and objects involved in interactions.
• To demonstrate that end-to-end regression with shared 3D convolutions and simple box connection during inference can achieve high speed and competitive accuracy.

Proposed method

• The STAR model uses a 3D ConvNet backbone with shared features to jointly regress spatial action boxes and temporal durations in an end-to-end manner.
• Spatial regression produces a single bounding box enclosing all interacting objects, rather than individual person-centric boxes.
• Temporal regression predicts the start and end timestamps of the action, enabling direct temporal localization without sliding windows or heuristic linking.
• During inference, the model connects the regressed spatial boxes within the predicted temporal duration using a simple IoU-based heuristic, avoiding complex optimization.
• The model is trained with RGB input only, using 3D convolutions to capture motion context, eliminating the need for optical flow.
• The architecture is efficient and supports variable spatial and temporal resolutions, with optimal performance at 25 fps and 176×176 resolution.

Experimental results

Research questions

• RQ1 How can spatio-temporal action detection be effectively extended to actions involving multiple interacting objects, rather than single-person actions?
• RQ2 Can a unified, end-to-end model with simultaneous spatial and temporal regression outperform pipeline-based methods that rely on separate object detection and tracking?
• RQ3 What is the impact of using only RGB input with 3D convolutions on detection accuracy and inference speed compared to two-stream models with optical flow?
• RQ4 How does the performance of the proposed model compare to state-of-the-art methods on standard benchmarks like UCF101-24 when applied to person-centric actions?
• RQ5 Can a simple heuristic for connecting spatial boxes within a predicted temporal window replace complex post-processing while maintaining accuracy?

Key findings

• The STAR model achieves a video mAP@0.5 of 53.0% on the UCF101-24 benchmark using only RGB input, demonstrating strong performance without optical flow or external detectors.
• The model achieves an inference speed of 250 fps on a single TITAN X (Pascal), significantly outpacing previous methods that operate below 40 fps.
• The model shows competitive results on UCF101-24 despite not being explicitly designed for person-centric actions, indicating strong generalization to standard benchmarks.
• The proposed something-STAR dataset contains 3207 videos with 2293 for training and 914 for testing, annotated with multi-object interaction tubes.
• The model’s frame mAP@0.5 is lower than its video mAP, indicating strong temporal localization ability despite weaker frame-level detection.
• The ablation study confirms that using a pre-trained classification backbone fine-tuned on something-something improves performance over using Sports-1M weights.