[Paper Review] Unsupervised Domain Adaptation for Video Transformers in Action Recognition

Victor G. Turrisi da Costa, Giacomo Zara+5 more|arXiv (Cornell University)|Jul 26, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes UDAVT, a novel unsupervised domain adaptation framework for video action recognition that leverages a spatio-temporal transformer backbone and a new information bottleneck-inspired domain alignment loss. By jointly fine-tuning the spatial and temporal transformers with domain-invariant feature learning, UDAVT achieves state-of-the-art performance on HMDB→UCF and the more challenging Kinetics→NEC-Drone benchmarks, demonstrating robustness to significant domain shift.

ABSTRACT

Over the last few years, Unsupervised Domain Adaptation (UDA) techniques have acquired remarkable importance and popularity in computer vision. However, when compared to the extensive literature available for images, the field of videos is still relatively unexplored. On the other hand, the performance of a model in action recognition is heavily affected by domain shift. In this paper, we propose a simple and novel UDA approach for video action recognition. Our approach leverages recent advances on spatio-temporal transformers to build a robust source model that better generalises to the target domain. Furthermore, our architecture learns domain invariant features thanks to the introduction of a novel alignment loss term derived from the Information Bottleneck principle. We report results on two video action recognition benchmarks for UDA, showing state-of-the-art performance on HMDB$\leftrightarrow$UCF, as well as on Kinetics$ ightarrow$NEC-Drone, which is more challenging. This demonstrates the effectiveness of our method in handling different levels of domain shift. The source code is available at https://github.com/vturrisi/UDAVT.

Motivation & Objective

• To address the challenge of domain shift in video action recognition, where models trained on one domain (e.g., HMDB) underperform on a different but related domain (e.g., UCF or NEC-Drone).
• To extend unsupervised domain adaptation (UDA) techniques—previously well-developed for images—into the more complex video domain with temporal dynamics.
• To develop a robust, end-to-end trainable UDA framework that improves generalization by learning domain-invariant representations in video transformers.
• To evaluate the effectiveness of the proposed method on standard UDA benchmarks for action recognition, including challenging cross-dataset settings.

Proposed method

• The method employs a two-phase training strategy: first, the full transformer is fine-tuned on source data only; then, the spatial transformer is frozen, and only the temporal transformer is further fine-tuned with a novel domain alignment loss.
• The core innovation is a new domain alignment loss derived from the Information Bottleneck (IB) principle, which encourages the model to retain only the information relevant to the action class while discarding domain-specific variations.
• The loss is applied during the second phase of training, using pseudo-labels from the source model on target data to align the feature distributions across domains.
• The architecture uses a shared encoder based on STAM (Spatio-Temporal Attention Module), combining a Vision Transformer (ViT) for spatial feature extraction and a multi-layer transformer for temporal aggregation.
• A momentum-updated queue of target features is used to stabilize the training of the domain alignment component, improving feature consistency.
• The method is evaluated using both supervised and unsupervised settings, with ablation studies on the role of the queue and pseudo-label quality.

Experimental results

Research questions

• RQ1 Can a transformer-based video model be effectively adapted to a new domain without labeled target data, using only source data and unlabeled target data?
• RQ2 How effective is an information bottleneck-based loss in aligning domain-invariant features for video action recognition compared to existing UDA losses?
• RQ3 Does the proposed two-phase training strategy—first pre-training on source data, then fine-tuning the temporal stream with domain alignment—improve model generalization under domain shift?
• RQ4 How does the quality of target pseudo-labels affect the performance of the domain adaptation process?
• RQ5 Can the proposed method outperform existing self-supervised contrastive learning methods (e.g., SimCLR, VICReg) when re-purposed for domain adaptation in video action recognition?

Key findings

• UDAVT achieves state-of-the-art performance on the HMDB→UCF benchmark, outperforming previous methods by a significant margin.
• On the more challenging Kinetics→NEC-Drone benchmark, UDAVT achieves 65.3% accuracy, a performance boost of over 16% compared to the second-best method.
• The ablation study confirms that both the momentum queue and high-quality pseudo-labels are critical for optimal performance, especially in the unsupervised setting.
• The method outperforms re-purposed self-supervised contrastive methods like SimCLR and VICReg, which were adapted for domain adaptation but still underperform compared to the proposed IB-based loss.
• Visual attention maps show that UDAVT improves spatio-temporal localization, with the model focusing more accurately on action-relevant frames in the sequence.
• The results demonstrate that the IB-based loss effectively reduces domain shift by learning disentangled representations that preserve class information while discarding domain-specific noise.