[Paper Review] Diversifying Spatial-Temporal Perception for Video Domain Generalization

Kun-Yu Lin, Jia-Run Du+3 more|arXiv (Cornell University)|Oct 27, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes the Spatial-Temporal Diversification Network (STDN) to improve video domain generalization by enhancing diversity in spatial and temporal feature learning. By introducing a Spatial Grouping Module and a Spatial-Temporal Relation Module, STDN discovers domain-invariant cues such as moving objects (e.g., basketballs) over static domain-specific cues (e.g., backboards), achieving state-of-the-art performance on three benchmarks including two new ones.

ABSTRACT

Video domain generalization aims to learn generalizable video classification models for unseen target domains by training in a source domain. A critical challenge of video domain generalization is to defend against the heavy reliance on domain-specific cues extracted from the source domain when recognizing target videos. To this end, we propose to perceive diverse spatial-temporal cues in videos, aiming to discover potential domain-invariant cues in addition to domain-specific cues. We contribute a novel model named Spatial-Temporal Diversification Network (STDN), which improves the diversity from both space and time dimensions of video data. First, our STDN proposes to discover various types of spatial cues within individual frames by spatial grouping. Then, our STDN proposes to explicitly model spatial-temporal dependencies between video contents at multiple space-time scales by spatial-temporal relation modeling. Extensive experiments on three benchmarks of different types demonstrate the effectiveness and versatility of our approach.

Motivation & Objective

• To address the challenge of video domain generalization where models overfit to domain-specific cues in the source domain.
• To reduce reliance on static, easily fitting domain-specific cues (e.g., backboards) that fail in unseen target domains.
• To discover and leverage domain-invariant cues (e.g., dynamic objects like basketballs) through enhanced spatial-temporal feature diversity.
• To improve model generalization by explicitly modeling spatial-temporal dependencies across multiple scales.
• To develop a unified framework that enhances feature diversity from both spatial and temporal dimensions.

Proposed method

• Proposes a Spatial Grouping Module (SGM) that applies a clustering-like process to extract diverse spatial cues from individual frames, improving spatial feature diversity.
• Introduces a Spatial-Temporal Relation Module (TRM) that models dependencies between video contents at multiple space-time scales, enhancing temporal and spatial relation diversity.
• Employs a relation discrimination loss $L_{\mathrm{rel}}$ to maximize the difference between temporal relation features at different time scales, preventing feature collapse.
• Uses normalized Mean Square Error (MSE) to quantitatively measure diversity in temporal relation features, with higher values indicating greater diversity.
• Applies t-SNE and Davies-Bouldin Index to evaluate spatial clustering quality, showing improved cluster separation with SGM.
• Utilizes Grad-CAM visualization to compare attention maps, demonstrating that STDN focuses on diverse, including domain-invariant, cues like moving basketballs.

Figure 1: Video classification models suffer from the misguidance of domain-specific cues when generalizing to unseen domains. As shown in the figure, in the source domain, the static backboard provides a clearer cue compared with the blurred basketball in motion, thus prevailing video classificatio

Experimental results

Research questions

• RQ1 Can enhancing spatial and temporal feature diversity reduce overfitting to domain-specific cues in video domain generalization?
• RQ2 Does modeling spatial-temporal dependencies across multiple scales improve generalization to unseen domains?
• RQ3 Can a clustering-like spatial grouping mechanism extract more diverse and discriminative spatial cues from video frames?
• RQ4 To what extent does the proposed relation discrimination loss improve feature diversity across time scales?
• RQ5 Can the model learn to rely on dynamic, domain-invariant cues (e.g., moving objects) instead of static, domain-specific cues (e.g., backboards) for better generalization?

Key findings

• STDN achieves state-of-the-art performance on three benchmarks, including two newly designed ones, demonstrating its effectiveness and versatility.
• The proposed relation discrimination loss $L_{\mathrm{rel}}$ significantly increases the normalized MSE between temporal relation features at different scales, indicating improved diversity.
• t-SNE visualization shows that the Spatial Grouping Module produces better-separated spatial clusters than average pooling, with a lower Davies-Bouldin Index.
• Grad-CAM visualization confirms that STDN attends to diverse cues, including dynamic basketballs, while the baseline model relies on static backboards.
• The model generalizes better to unseen domains where domain-specific cues (e.g., backboards) are occluded, due to its ability to leverage domain-invariant cues.
• Quantitative analysis shows that STDN’s feature diversity is higher than baselines, especially when combining SGM and TRM with the relation loss.

Figure 2: An overview of our proposed Spatial-Temporal Diversification Network (STDN). We use a video of $N=5$ segments with $K=4$ spatial groups for example. After backbone feature extraction, our STDN extracts spatial cues of $K$ types for each frame by the Spatial Grouping Module, enriching the d