[Paper Review] LaT: Latent Translation with Cycle-Consistency for Video-Text Retrieval

Jinbin Bai, Chunhui Liu+5 more|arXiv (Cornell University)|Jul 11, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes LaT, a latent translation framework with cycle-consistency to bridge the modality gap in video-text retrieval without relying on a shared joint embedding space. By learning bidirectional translations between visual and textual latent spaces using learnable query-guided decoders and enforcing cycle consistency, LaT preserves intra-modal information better than joint space methods, achieving state-of-the-art performance on MSR-VTT, MSVD, and DiDeMo.

ABSTRACT

Video-text retrieval is a class of cross-modal representation learning problems, where the goal is to select the video which corresponds to the text query between a given text query and a pool of candidate videos. The contrastive paradigm of vision-language pretraining has shown promising success with large-scale datasets and unified transformer architecture, and demonstrated the power of a joint latent space. Despite this, the intrinsic divergence between the visual domain and textual domain is still far from being eliminated, and projecting different modalities into a joint latent space might result in the distorting of the information inside the single modality. To overcome the above issue, we present a novel mechanism for learning the translation relationship from a source modality space $\mathcal{S}$ to a target modality space $\mathcal{T}$ without the need for a joint latent space, which bridges the gap between visual and textual domains. Furthermore, to keep cycle consistency between translations, we adopt a cycle loss involving both forward translations from $\mathcal{S}$ to the predicted target space $\mathcal{T'}$, and backward translations from $\mathcal{T'}$ back to $\mathcal{S}$. Extensive experiments conducted on MSR-VTT, MSVD, and DiDeMo datasets demonstrate the superiority and effectiveness of our LaT approach compared with vanilla state-of-the-art methods.

Motivation & Objective

• To address the intrinsic modality gap between visual and textual domains in video-text retrieval.
• To mitigate distortion of intra-modal information caused by projecting different modalities into a shared joint latent space.
• To learn a robust translation mechanism between visual and textual latent spaces without relying on a unified embedding space.
• To enforce cycle consistency in bidirectional translations to improve alignment and feature fidelity.
• To achieve better cross-modal retrieval performance through modality-specific representation preservation.

Proposed method

• Proposes a dual-decoder architecture to translate between visual and textual latent spaces using learnable query parameters $Q_G$ and $Q_F$.
• Introduces forward translation $G: \mathcal{S} \to \mathcal{T}'$ and backward translation $F: \mathcal{T}' \to \mathcal{S}$ to enable cross-modal mapping.
• Enforces cycle consistency via constraints $v \approx G(F(v))$ and $t \approx F(G(t))$ to ensure consistency after double translation.
• Uses a cycle loss to optimize both forward and backward translation paths, improving alignment and reducing distribution shift.
• Employs contrastive learning with CLIP-style pretraining, but replaces joint space projection with modality-specific translation.
• Leverages dimensionally reduced visualizations and cosine similarity analysis to validate improved modality fusion.

Experimental results

Research questions

• RQ1 Can a joint latent space be avoided while still achieving strong cross-modal alignment in video-text retrieval?
• RQ2 Does cycle-consistent latent translation preserve intra-modal information better than standard contrastive learning in a shared space?
• RQ3 Can learnable query-guided decoders improve the fidelity of cross-modal translation between vision and language?
• RQ4 How does the proposed method compare to SOTA baselines in terms of retrieval accuracy across standard benchmarks?
• RQ5 What is the impact of cycle consistency on reducing modality gap and improving retrieval performance?

Key findings

• LaT achieves state-of-the-art performance on MSR-VTT, MSVD, and DiDeMo datasets, outperforming vanilla SOTA methods.
• Cosine similarity between matched image-text pairs in translated spaces (GT and FV) is consistently higher than in original spaces, indicating better alignment.
• The cosine similarity between translated embeddings of the same semantic concept (e.g., 'Apple') is significantly higher than between different concepts, confirming effective fusion.
• Visualization in Figure 4 shows that LaT better merges textual and visual embeddings than CLIP, with tighter clustering of matched pairs.
• The method reduces the modality gap, as evidenced by higher similarity between a text description and its corresponding image after translation (e.g., 0.58 for 'Running man' vs. 0.27 in CLIP).
• Cycle consistency loss improves translation fidelity, as shown by consistent reconstruction of original embeddings after double translation.