[Paper Review] A Structured Model For Action Detection

Yubo Zhang, Pavel Tokmakov+2 more|arXiv (Cornell University)|Dec 9, 2018

Human Pose and Action Recognition60 references4 citations

TL;DR

This paper proposes a structured deep learning model for video action detection that integrates long-term temporal modeling via actor tracking and relational reasoning through graph convolutional networks to capture human-object and human-human interactions. By incorporating domain knowledge into the model architecture—using I3D features, tubelet-based tracking, and a soft relation graph—it achieves 22.2 mAP on the AVA dataset, outperforming the previous state of the art by 4.8%.

ABSTRACT

A dominant paradigm for learning-based approaches in computer vision is training generic models, such as ResNet for image recognition, or I3D for video understanding, on large datasets and allowing them to discover the optimal representation for the problem at hand. While this is an obviously attractive approach, it is not applicable in all scenarios. We claim that action detection is one such challenging problem - the models that need to be trained are large, and labeled data is expensive to obtain. To address this limitation, we propose to incorporate domain knowledge into the structure of the model, simplifying optimization. In particular, we augment a standard I3D network with a tracking module to aggregate long term motion patterns, and use a graph convolutional network to reason about interactions between actors and objects. Evaluated on the challenging AVA dataset, the proposed approach improves over the I3D baseline by 5.5% mAP and over the state-of-the-art by 4.8% mAP.

Motivation & Objective

• Address the challenge of action detection in videos where large models and expensive annotations limit performance.
• Improve action detection by modeling long-term temporal dependencies beyond short-term motion patterns.
• Enhance recognition of complex actions requiring interaction reasoning between actors and objects.
• Reduce reliance on expensive object annotations by using weakly-supervised object detection and category-agnostic object proposals.
• Demonstrate that architectural inductive biases—such as tracking and relational graphs—significantly improve performance on action detection tasks.

Proposed method

• Extract I3D features from video frames to encode spatiotemporal representations.
• Use a modified, category-agnostic object detection framework (based on He et al. [17]) to localize actors and objects without requiring category-specific annotations.
• Apply a heuristic tracker to generate 3-second tubelets—temporal sequences of bounding boxes—for each actor to model long-term motion.
• Construct an actor-centric graph where nodes represent actors and objects, and edges encode potential human-object or human-human interactions.
• Pool I3D features over tubelets and object proposals to form node representations, then apply graph convolutional networks to reason about relational features.
• Train a classifier on edge-level features to predict action labels, enabling end-to-end learning with weak supervision on object locations.

Experimental results

Research questions

• RQ1 Can explicit modeling of long-term actor motion through tubelet tracking improve action detection performance on long-duration actions?
• RQ2 How does incorporating relational reasoning between actors and objects enhance recognition of interaction-based actions?
• RQ3 To what extent can weakly-supervised object detection reduce dependency on costly instance-level annotations in action detection?
• RQ4 Does architectural inductive bias—such as tracking and graph structures—lead to better generalization than end-to-end learned representations alone?
• RQ5 How do the proposed components compare in isolation and combination in terms of mAP gain on the AVA benchmark?

Key findings

• The proposed model achieves 22.2 mAP on the AVA validation set, representing a 4.8% improvement over the previous state of the art (ACRN).
• The model shows the largest performance gains on actions requiring long-term temporal modeling, such as 'drive', 'play musical instrument', and 'hand clap'.
• The use of actor tubelets and soft relation graphs leads to a 5.5% mAP improvement over the I3D baseline, demonstrating the effectiveness of long-term and relational modeling.
• Qualitative analysis confirms that the model correctly captures spatiotemporal context: it distinguishes 'eat' from 'hold' by integrating fork features, and correctly identifies 'fight' by modeling dual-actor interactions.
• On UCF-101-24, the model achieves 77.9 mAP, outperforming the I3D baseline (72.0 mAP), though it remains below the SOTA due to the use of I3D instead of S3D as backbone.
• Ablation studies confirm that both tubelet tracking and relational graph components are essential, with the highest gains observed in actions involving complex motion and object interaction.