[Paper Review] Unsupervised Domain Adaptation for Spatio-Temporal Action Localization

Nakul Agarwal, Yi‐Ting Chen+2 more|arXiv (Cornell University)|Oct 19, 2020

Human Pose and Action Recognition22 references4 citations

TL;DR

This paper proposes an end-to-end unsupervised domain adaptation framework for spatio-temporal action localization that reduces domain shift by jointly adapting spatial and temporal features at image-level and instance-level. By integrating three adversarial adaptation modules—temporal and spatial at image level, and temporal at instance level—on Faster R-CNN, the method achieves significant performance gains across UCF-Sports, UCF-101, and JHMDB datasets, especially in cross-domain settings with visual and temporal discrepancies.

ABSTRACT

Spatio-temporal action localization is an important problem in computer vision that involves detecting where and when activities occur, and therefore requires modeling of both spatial and temporal features. This problem is typically formulated in the context of supervised learning, where the learned classifiers operate on the premise that both training and test data are sampled from the same underlying distribution. However, this assumption does not hold when there is a significant domain shift, leading to poor generalization performance on the test data. To address this, we focus on the hard and novel task of generalizing training models to test samples without access to any labels from the latter for spatio-temporal action localization by proposing an end-to-end unsupervised domain adaptation algorithm. We extend the state-of-the-art object detection framework to localize and classify actions. In order to minimize the domain shift, three domain adaptation modules at image level (temporal and spatial) and instance level (temporal) are designed and integrated. We design a new experimental setup and evaluate the proposed method and different adaptation modules on the UCF-Sports, UCF-101 and JHMDB benchmark datasets. We show that significant performance gain can be achieved when spatial and temporal features are adapted separately, or jointly for the most effective results.

Motivation & Objective

• Address the challenge of poor generalization in spatio-temporal action localization due to domain shift between training and test data.
• Develop a novel unsupervised domain adaptation framework for action localization where no target labels are available.
• Design and integrate multiple domain adaptation modules to jointly align spatial and temporal features across domains.
• Establish a new experimental setup to evaluate domain adaptation in spatio-temporal action localization, as existing benchmarks lack such evaluation protocols.
• Demonstrate that separate and joint adaptation of spatial and temporal features significantly improves localization performance under domain shift.

Proposed method

• Extend the Faster R-CNN framework to support spatio-temporal action localization with I3D backbones for video feature extraction.
• Introduce three domain adaptation modules: (1) image-level temporal feature alignment, (2) instance-level temporal feature alignment, and (3) image-level spatial feature alignment.
• Use adversarial training with domain classifiers to learn domain-invariant features at both image and instance levels.
• Train the adaptation modules end-to-end with the detection head, enabling joint optimization of localization and domain alignment.
• Leverage global scene context in image-level temporal adaptation and actor/action dynamics in instance-level adaptation for improved feature generalization.
• Design a new evaluation protocol using cross-dataset adaptation (e.g., UCF-101 → JHMDB) to validate performance under domain shift.

Experimental results

Research questions

• RQ1 Can unsupervised domain adaptation effectively reduce domain shift in spatio-temporal action localization when no target annotations are available?
• RQ2 How do image-level and instance-level temporal feature alignment compare in improving localization performance?
• RQ3 What is the contribution of spatial feature alignment in reducing localization errors and improving proposal quality?
• RQ4 Does joint adaptation of spatial and temporal features yield better performance than individual adaptation?
• RQ5 How do visual and temporal differences between datasets (e.g., UCF-101 vs. JHMDB) affect model generalization, and can adaptation mitigate this?

Key findings

• The proposed method achieves significant performance gains on UCF-Sports, UCF-101, and JHMDB datasets under unsupervised domain adaptation settings, with mAP improvements over the baseline.
• Joint adaptation of spatial and temporal features yields the best performance, outperforming individual adaptation modules, especially in cases with high visual and temporal domain shift.
• Instance-level temporal adaptation reduces mislocalization and incorrect classification errors, indicating improved action dynamics modeling.
• Image-level spatial adaptation significantly reduces mislocalization errors by improving region proposal network (RPN) quality, directly enhancing localization accuracy.
• The model with all three adaptation modules achieves the highest number of correct detections and the lowest mislocalization rate in error analysis on top-1000 detections.
• The baseline model performs poorly on JHMDB despite large-scale training data (e.g., UCF-101), highlighting the impact of domain shift—especially for Walk and Basketball classes—while adaptation closes the gap significantly.