[Paper Review] Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignment

Sarah Ibrahimi, Xiaohang Sun+4 more|arXiv (Cornell University)|Jul 24, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes TEFAL, a text-conditioned feature alignment method that independently aligns audio and video representations to the text query using separate cross-attention blocks, avoiding suppression of relevant audio cues. It achieves state-of-the-art performance on four audio-rich text-to-video retrieval benchmarks, improving Recall@1 by over 4% over E-CLIP-SE on MSR-VTT.

ABSTRACT

Text-to-video retrieval systems have recently made significant progress by utilizing pre-trained models trained on large-scale image-text pairs. However, most of the latest methods primarily focus on the video modality while disregarding the audio signal for this task. Nevertheless, a recent advancement by ECLIPSE has improved long-range text-to-video retrieval by developing an audiovisual video representation. Nonetheless, the objective of the text-to-video retrieval task is to capture the complementary audio and video information that is pertinent to the text query rather than simply achieving better audio and video alignment. To address this issue, we introduce TEFAL, a TExt-conditioned Feature ALignment method that produces both audio and video representations conditioned on the text query. Instead of using only an audiovisual attention block, which could suppress the audio information relevant to the text query, our approach employs two independent cross-modal attention blocks that enable the text to attend to the audio and video representations separately. Our proposed method's efficacy is demonstrated on four benchmark datasets that include audio: MSR-VTT, LSMDC, VATEX, and Charades, and achieves better than state-of-the-art performance consistently across the four datasets. This is attributed to the additional text-query-conditioned audio representation and the complementary information it adds to the text-query-conditioned video representation.

Motivation & Objective

• To address the limitation of existing audio-enhanced text-to-video retrieval methods that suppress relevant audio cues by fusing audio and video before text alignment.
• To improve retrieval performance by learning audio and video representations that are explicitly conditioned on the text query, rather than jointly aligning audio and video.
• To demonstrate that text-conditioned audio and video representations provide complementary information that enhances retrieval accuracy beyond audiovisual fusion.
• To validate the effectiveness of TEFAL across diverse, large-scale video retrieval benchmarks with audio.

Proposed method

• TEFAL uses CLIP as a frozen text and video feature extractor, and AST as a frozen audio feature extractor to generate initial embeddings.
• It applies two independent cross-attention blocks: one for text-to-audio and one for text-to-video, enabling the text to attend to each modality separately.
• The text query acts as the query, while audio and video features serve as keys and values in each cross-attention mechanism.
• The resulting text-conditioned audio and video representations are added together to form a final audio-enhanced video representation.
• The method avoids joint audiovisual attention to preserve audio cues that may not be visually present, such as spoken words or off-screen sounds.
• For scalable inference, TEFAL is used to re-rank top-K results from a mean-pooled video retrieval baseline, reducing computational cost without significant performance drop.

Experimental results

Research questions

• RQ1 Does conditioning audio and video representations independently on the text query lead to better retrieval performance than joint audiovisual alignment?
• RQ2 Can text-conditioned audio representations improve retrieval when audio cues are not visually present, such as spoken numbers or off-screen sounds?
• RQ3 Is the performance gain from audio enhancement due to complementary information from audio and video, or simply from better modality alignment?
• RQ4 How does TEFAL scale in practice when applied to large video datasets with high inference cost?

Key findings

• TEFAL achieves a Recall@1 of 49.4% on MSR-VTT, which is over 4% higher than E-CLIP-SE, demonstrating state-of-the-art performance.
• On LSMDC, TEFAL improves Recall@1 from 20.9% (mean-pooling baseline) to 26.8%, showing consistent gains across datasets.
• The addition of text-conditioned audio representations improves performance by 2.5% over video-only baselines, confirming the value of audio in retrieval.
• Re-ranking with TEFAL on a reduced search space (top 10% from mean-pooling) results in only a negligible performance drop, proving scalability.
• Fusion methods like stacking audio and video embeddings or using a third cross-attention block underperform compared to simple addition, indicating that fusion is non-trivial.
• The ablation study confirms that separate cross-attention for audio and video is superior to audiovisual fusion, as it prevents suppression of relevant audio cues.