[Paper Review] Learning to Localize Actions from Moments

Fuchen Long, Ting Yao+4 more|arXiv (Cornell University)|Aug 31, 2020

Human Pose and Action Recognition51 references4 citations

TL;DR

This paper proposes Action Herald Networks (AherNet), a one-stage action localization framework that enables large-scale action localization using only trimmed action moments from a large set of categories (e.g., Kinetics-600) and temporal annotations from a small set of classes (e.g., ActivityNet v1.3). By leveraging a shared weight transfer function and adversarial context generation, AherNet achieves state-of-the-art performance even surpassing fully-supervised methods, with mAP of 24.43% on Kinetics-600 under 0.5 IoU threshold.

ABSTRACT

With the knowledge of action moments (i.e., trimmed video clips that each contains an action instance), humans could routinely localize an action temporally in an untrimmed video. Nevertheless, most practical methods still require all training videos to be labeled with temporal annotations (action category and temporal boundary) and develop the models in a fully-supervised manner, despite expensive labeling efforts and inapplicable to new categories. In this paper, we introduce a new design of transfer learning type to learn action localization for a large set of action categories, but only on action moments from the categories of interest and temporal annotations of untrimmed videos from a small set of action classes. Specifically, we present Action Herald Networks (AherNet) that integrate such design into an one-stage action localization framework. Technically, a weight transfer function is uniquely devised to build the transformation between classification of action moments or foreground video segments and action localization in synthetic contextual moments or untrimmed videos. The context of each moment is learnt through the adversarial mechanism to differentiate the generated features from those of background in untrimmed videos. Extensive experiments are conducted on the learning both across the splits of ActivityNet v1.3 and from THUMOS14 to ActivityNet v1.3. Our AherNet demonstrates the superiority even comparing to most fully-supervised action localization methods. More remarkably, we train AherNet to localize actions from 600 categories on the leverage of action moments in Kinetics-600 and temporal annotations from 200 classes in ActivityNet v1.3. Source code and data are available at \url{https://github.com/FuchenUSTC/AherNet}.

Motivation & Objective

• To enable action localization for thousands of categories without requiring full temporal annotations for all categories.
• To bridge the gap between action recognition on trimmed moments and action localization on untrimmed videos using a transfer learning paradigm.
• To address the challenge of simulating realistic background contexts for action moments during training.
• To develop a scalable, partially-supervised framework that reduces annotation cost while maintaining high localization accuracy.
• To demonstrate the feasibility of training a single model for large-scale action localization using minimal supervision from a small set of annotated classes.

Proposed method

• AherNet employs a one-stage action localization framework integrating weight transfer between classification and localization tasks.
• A shared weight transfer function maps network parameters from foreground segment classification to temporal action classification in untrimmed videos.
• Adversarial learning is used to generate realistic background features for action moments, distinguishing them from real untrimmed video backgrounds.
• The model is trained end-to-end using proposal loss, classification loss, and adversarial loss to optimize localization and context realism.
• Action moments are treated as synthetic contexts, enabling localization modeling without full temporal annotations.
• The framework is applied across splits of ActivityNet v1.3 and from THUMOS14 to ActivityNet v1.3, with extension to 600-class Kinetics-600.

Experimental results

Research questions

• RQ1 Can action localization be effectively trained using only trimmed action moments from a large set of categories and temporal annotations from a small set of classes?
• RQ2 How can a weight transfer function effectively bridge classification on action moments and localization on untrimmed videos?
• RQ3 Can adversarial learning generate realistic background features that mimic real untrimmed video backgrounds for action moment training?
• RQ4 To what extent can a partially-supervised model generalize to large-scale action categories without full supervision?
• RQ5 Does the proposed method outperform fully-supervised and weakly-supervised baselines in zero-shot or few-shot localization settings?

Key findings

• AherNet achieves a mAP of 24.43% on Kinetics-600 under 0.5 IoU threshold, a significant improvement from 14.18% baseline.
• The model outperforms the most recent weakly-supervised method by 3.9% and 5.8% under IoU thresholds of 0.5 and 0.75, respectively.
• AherNet surpasses several fully-supervised models that rely on full temporal annotations for all categories.
• The adversarial context generation module produces background features indistinguishable from real background features on both ANet-AM and ANet-FG sets.
• AherNet ∗, the fully-supervised variant, only exceeds AherNet in performance when more than 50% of temporal annotations are used, confirming the efficiency of the partial-supervision design.
• Qualitative results on Kinetics-600 show accurate temporal boundary predictions and effective modeling of action dynamics.