[Paper Review] Attentive Action and Context Factorization

Yang Wang, Vinh Cao Trần+3 more|arXiv (Cornell University)|Apr 10, 2019

Human Pose and Action Recognition27 references5 citations

TL;DR

This paper proposes an attentive action and context factorization method that uses conjugate video samples—contextually similar but action-free clips—to train a weakly supervised attention mechanism for spatial-temporal localization of human actions. The approach learns to disentangle action and context features via dual spatiotemporal attention maps, improving action recognition accuracy and interpretability without requiring detailed annotations.

ABSTRACT

We propose a method for human action recognition, one that can localize the spatiotemporal regions that `define' the actions. This is a challenging task due to the subtlety of human actions in video and the co-occurrence of contextual elements. To address this challenge, we utilize conjugate samples of human actions, which are video clips that are contextually similar to human action samples but do not contain the action. We introduce a novel attentional mechanism that can spatially and temporally separate human actions from the co-occurring contextual factors. The separation of the action and context factors is weakly supervised, eliminating the need for laboriously detailed annotation of these two factors in training samples. Our method can be used to build human action classifiers with higher accuracy and better interpretability. Experiments on several human action recognition datasets demonstrate the quantitative and qualitative benefits of our approach.

Motivation & Objective

• To address the challenge of localizing spatiotemporal regions defining human actions in videos, where actions are subtle and co-occur with contextual elements.
• To reduce reliance on expensive, detailed annotations by leveraging conjugate samples—contextually similar clips without the target action.
• To improve model interpretability by explicitly separating action and context representations through attention mechanisms.
• To develop a method that enhances classification performance while providing visual grounding of action and context components.

Proposed method

• Uses paired training data consisting of action videos and their conjugate samples (contextually similar but action-free).
• Employs a dual-branch attention mechanism to generate separate spatiotemporal attention maps for action and context components.
• Applies spatial and temporal attention to selectively pool features from relevant video regions, enabling disentangled feature extraction.
• Optimizes a joint objective: minimizing classification loss, maximizing similarity between context features of action and conjugate samples, and minimizing similarity between action features.
• Utilizes 3D CNN backbones (e.g., 3D-Res34-RGB) for feature extraction and applies global average pooling on attention-weighted features.
• Adapts the framework to still images using image completion and blending to generate conjugate samples for Pascal VOC2012.

Experimental results

Research questions

• RQ1 Can a weakly supervised attention mechanism effectively localize action-relevant spatiotemporal regions in videos without detailed annotations?
• RQ2 How well can the model disentangle action and context representations using only conjugate samples?
• RQ3 To what extent does the proposed method improve action recognition accuracy and interpretability compared to baseline attention mechanisms?
• RQ4 How effective is the method on short-duration video clips, such as those in the HACS dataset?
• RQ5 Can the framework be generalized to still images for action and context localization?

Key findings

• The proposed ActX method achieves a mean average precision (mAP) of 80.2% on the Pascal VOC2012 dataset, outperforming the prior state-of-the-art of 75.2%.
• On the HACS-30 dataset, ActX achieves higher mAP and mAcc than baseline attention mechanisms, though performance gains are smaller due to short video duration.
• The method demonstrates improved interpretability through attention maps that clearly localize human-object interactions (action) and surrounding scenes (context).
• Experiments on ActionThread and Hollywood2 datasets confirm consistent performance gains over existing attention-based models.
• The attention maps for action and context are visually coherent, with action maps focusing on human-object interactions and context maps on background and scene elements.
• The use of conjugate samples enables effective weak supervision, eliminating the need for pixel-level or bounding box annotations of action and context.