[Paper Review] BiC-Net: Learning Efficient Spatio-Temporal Relation for Text-Video Retrieval

Ning Han, Jingjing Chen+4 more|arXiv (Cornell University)|Oct 29, 2021

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes BiC-Net, a dual-branch transformer-based framework that jointly models global temporal features and fine-grained spatio-temporal relations in videos for text-video retrieval. By integrating a Spatio-Temporal Residual Transformer (SRT) for local relations and a multi-layer transformer for global context, BiC-Net achieves state-of-the-art performance, outperforming prior methods by up to 18.5% in R@10 accuracy on MSR-VTT 1k-A with lower computational cost.

ABSTRACT

The task of text-video retrieval aims to understand the correspondence between language and vision, has gained increasing attention in recent years. Previous studies either adopt off-the-shelf 2D/3D-CNN and then use average/max pooling to directly capture spatial features with aggregated temporal information as global video embeddings, or introduce graph-based models and expert knowledge to learn local spatial-temporal relations. However, the existing methods have two limitations: 1) The global video representations learn video temporal information in a simple average/max pooling manner and do not fully explore the temporal information between every two frames. 2) The graph-based local video representations are handcrafted, it depends heavily on expert knowledge and empirical feedback, which may not be able to effectively mine the higher-level fine-grained visual relations. These limitations result in their inability to distinguish videos with the same visual components but with different relations. To solve this problem, we propose a novel cross-modal retrieval framework, Bi-Branch Complementary Network (BiC-Net), which modifies transformer architecture to effectively bridge text-video modalities in a complementary manner via combining local spatial-temporal relation and global temporal information. Specifically, local video representations are encoded using multiple transformer blocks and additional residual blocks to learn spatio-temporal relation features, calling the module a Spatio-Temporal Residual transformer (SRT). Meanwhile, Global video representations are encoded using a multi-layer transformer block to learn global temporal features. Finally, we align the spatio-temporal relation and global temporal features with the text feature on two embedding spaces for cross-modal text-video retrieval.

Motivation & Objective

• To address the limitation of existing text-video retrieval models in distinguishing videos with identical visual components but different spatial-temporal relations.
• To overcome the reliance on handcrafted graphs or simple pooling in modeling local spatio-temporal relations, which limits fine-grained visual relation understanding.
• To develop a unified framework that complements global temporal representations with fine-grained local relational features for improved cross-modal alignment.
• To reduce computational overhead while maintaining high performance through an efficient transformer-based architecture.

Proposed method

• BiC-Net employs a dual-branch architecture: one branch uses multiple transformer blocks with residual connections to learn local spatio-temporal relation features, termed the Spatio-Temporal Residual Transformer (SRT).
• The second branch uses a multi-layer transformer to encode global temporal features from the entire video sequence.
• Local and global video representations are aligned with text features in two separate embedding spaces for cross-modal retrieval.
• The model leverages self-attention mechanisms to capture long-range dependencies and relational cues between objects across frames.
• The SRT module enhances feature learning by combining residual connections with multi-head attention to preserve spatial and temporal context.
• The framework is trained end-to-end using contrastive loss to optimize cross-modal similarity between text and video embeddings.

Experimental results

Research questions

• RQ1 Can modeling fine-grained spatio-temporal relations improve text-video retrieval beyond global video representations?
• RQ2 How does a learnable, attention-based local relation modeling approach compare to handcrafted graph-based methods in capturing visual interactions?
• RQ3 Can a dual-branch architecture that combines local relational and global temporal features achieve better retrieval performance than single-modality modeling?
• RQ4 To what extent does the proposed method reduce computational cost while maintaining or improving accuracy compared to existing transformer-based models?

Key findings

• BiC-Net achieves 85.6% text-to-video R@10 accuracy on the MSR-VTT 1k-A test set, representing a 18.5% improvement over the MMT baseline.
• The model with one transformer layer outperforms DualEncoding in both accuracy and efficiency, with fewer parameters and lower FLOPs.
• Qualitative results show that BiC-Net successfully retrieves videos with complex, fine-grained actions such as 'cutting open a crab' and 'applying eye shadow to the right eye', while the variant without relation modeling fails to distinguish relevant from similar but incorrect videos.
• Attention visualization confirms that the model focuses on semantically relevant regions, such as 'man + crab' and 'woman + makeup brush', demonstrating effective learning of relational cues.
• The ablation study indicates that removing the spatio-temporal relation branch significantly degrades performance, proving its necessity for distinguishing videos with similar components but different interactions.
• The model maintains strong performance across three benchmarks—MSR-VTT, MSVD, and YouCook2—demonstrating generalization and robustness.