[Paper Review] Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language

Otniel-Bogdan Mercea, Lukas Riesch+2 more|arXiv (Cornell University)|Mar 7, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes AVCA, a cross-modal attention model for audio-visual generalised zero-shot learning (GZSL) that aligns audio-visual embeddings with textual label embeddings. By leveraging cross-attention between modalities and a novel loss function, AVCA achieves state-of-the-art performance on three new benchmarks—VGGSound, UCF101, and ActivityNet—outperforming prior methods by significant margins, especially on unseen classes.

ABSTRACT

Learning to classify video data from classes not included in the training data, i.e. video-based zero-shot learning, is challenging. We conjecture that the natural alignment between the audio and visual modalities in video data provides a rich training signal for learning discriminative multi-modal representations. Focusing on the relatively underexplored task of audio-visual zero-shot learning, we propose to learn multi-modal representations from audio-visual data using cross-modal attention and exploit textual label embeddings for transferring knowledge from seen classes to unseen classes. Taking this one step further, in our generalised audio-visual zero-shot learning setting, we include all the training classes in the test-time search space which act as distractors and increase the difficulty while making the setting more realistic. Due to the lack of a unified benchmark in this domain, we introduce a (generalised) zero-shot learning benchmark on three audio-visual datasets of varying sizes and difficulty, VGGSound, UCF, and ActivityNet, ensuring that the unseen test classes do not appear in the dataset used for supervised training of the backbone deep models. Comparing multiple relevant and recent methods, we demonstrate that our proposed AVCA model achieves state-of-the-art performance on all three datasets. Code and data are available at \url{https://github.com/ExplainableML/AVCA-GZSL}.

Motivation & Objective

• Address the challenge of classifying video data from unseen classes in audio-visual zero-shot learning (ZSL), where models must generalize beyond seen training classes.
• Overcome limitations in existing ZSL benchmarks that suffer from class overlap between training, validation, and test sets, leading to overfitting and inflated performance.
• Develop a unified, realistic benchmark for audio-visual GZSL using three diverse datasets—VGGSound, UCF101, and ActivityNet—ensuring no class leakage between training and test sets.
• Design a computationally efficient, multi-stream model that fuses audio and visual features using cross-attention while preserving unimodal information.
• Demonstrate that multi-modal representations significantly outperform unimodal baselines in zero-shot generalization, especially for unseen classes.

Proposed method

• Propose AVCA, a multi-stream neural network with separate audio and visual encoders that exchange information via cross-attention mechanisms to build joint representations.
• Use pre-extracted audio and visual features from self-supervised SeLaVi models to avoid information leakage from supervised pre-training to the ZSL task.
• Train the model using a combination of triplet-based contrastive losses and a regularization loss ($l_c$) that preserves salient unimodal features in the fused representation.
• Align the final audio-visual embedding with textual label embeddings via cross-attention, enabling zero-shot prediction by finding the closest matching label embedding.
• Introduce a novel training objective that encourages the model to attend to relevant cross-modal cues while maintaining discriminative unimodal information.
• Apply harmonic mean (HM) and zero-shot learning (ZSL) accuracy as evaluation metrics in a generalised ZSL setting where all training classes are present in the test search space as distractors.

Experimental results

Research questions

• RQ1 Can cross-modal attention between audio and visual streams improve zero-shot generalization in video classification when no training examples of the test class are available?
• RQ2 Does leveraging textual label embeddings in conjunction with audio-visual representations lead to better performance on unseen classes compared to unimodal or non-attention-based multi-modal methods?
• RQ3 How does the proposed AVCA model compare to state-of-the-art methods in generalised zero-shot learning (GZSL) on real-world audio-visual datasets with no class overlap between training and test sets?
• RQ4 To what extent does the inclusion of a regularization loss that preserves unimodal information improve the robustness and performance of multi-modal ZSL models?
• RQ5 Can a unified benchmark for audio-visual GZSL be constructed from existing datasets like VGGSound, UCF101, and ActivityNet that ensures realistic evaluation conditions without data leakage?

Key findings

• AVCA achieves a harmonic mean (HM) of 6.31% and zero-shot learning (ZSL) accuracy of 6.00% on the VGGSound-GZSL benchmark, outperforming AVGZSLNet (HM: 5.83%, ZSL: 5.28%) and DEVISE (HM: 5.00%, ZSL: 4.50%).
• On UCF-GZSL, AVCA achieves HM: 27.15% and ZSL: 20.01%, surpassing AVGZSLNet (HM: 18.05%, ZSL: 13.65%) and DEVISE (HM: 15.00%, ZSL: 12.00%).
• On ActivityNet-GZSL, AVCA achieves HM: 12.13% and ZSL: 9.13%, significantly outperforming AVGZSLNet (HM: 8.30%, ZSL: 6.39%) and DEVISE (HM: 7.00%, ZSL: 5.50%).
• Ablation studies show that removing cross-attention reduces performance, while adding the $l_c$ loss and cross-attention improves results over AVGZSLNet, proving the effectiveness of the proposed components.
• Training with both audio and visual branches yields higher seen (S) and unseen (U) performance than using either modality alone, with HM increasing from 7.53% (visual-only) to 12.13% on ActivityNet-GZSL.
• AVCA has 1.69M parameters and 2.36 GFLOPS, making it computationally efficient and comparable to AVGZSLNet (1.32M parameters, 1.38 GFLOPS), while achieving significantly better performance across all benchmarks.