[Paper Review] AudioCLIP: Extending CLIP to Image, Text and Audio

Andrey Guzhov, Federico Raue+2 more|arXiv (Cornell University)|Jun 24, 2021

Music and Audio Processing4 citations

TL;DR

This paper proposes AudioCLIP, a tri-modal extension of the CLIP model that integrates an ESResNeXt-based audio encoder with CLIP's vision-text contrastive learning framework, enabling zero-shot audio classification and cross-modal querying. The model achieves state-of-the-art accuracy of 90.07% on UrbanSound8K and 97.15% on ESC-50, while setting new zero-shot baselines of 68.78% and 69.40% respectively.

ABSTRACT

In the past, the rapidly evolving field of sound classification greatly benefited from the application of methods from other domains. Today, we observe the trend to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models. In this work, we present an extension of the CLIP model that handles audio in addition to text and images. Our proposed model incorporates the ESResNeXt audio-model into the CLIP framework using the AudioSet dataset. Such a combination enables the proposed model to perform bimodal and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen datasets in a zero-shot inference fashion. AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task, out-performing other approaches by reaching accuracies of 90.07% on the UrbanSound8K and 97.15% on the ESC-50 datasets. Further it sets new baselines in the zero-shot ESC-task on the same datasets (68.78% and 69.40%, respectively). Finally, we also assess the cross-modal querying performance of the proposed model as well as the influence of full and partial training on the results. For the sake of reproducibility, our code is published.

Motivation & Objective

• To extend the CLIP framework to include audio as a third modality, enabling joint text, image, and audio representation learning.
• To enable zero-shot generalization in audio classification by leveraging contrastive learning on the AudioSet dataset.
• To evaluate cross-modal querying capabilities between text, images, and audio in both zero-shot and fine-tuned settings.
• To investigate the impact of full versus partial training on model performance across classification and querying tasks.
• To establish new state-of-the-art results in environmental sound classification and provide a reproducible, open-source implementation.

Proposed method

• Integrates the ESResNeXt audio model into the CLIP framework as the audio encoder, replacing the original vision encoder for audio input.
• Trains the model using contrastive learning on the AudioSet dataset, aligning audio embeddings with corresponding text and image embeddings.
• Applies a two-stage training strategy: first, pre-training the audio head using contrastive loss with frozen text and image encoders; second, full end-to-end fine-tuning.
• Uses a dual-branch contrastive loss to maximize similarity between matching text-audio, image-audio, and text-image pairs while minimizing similarity between non-matching pairs.
• Supports zero-shot inference by computing cosine similarity between query embeddings (text, image, or audio) and class-specific embedding vectors.
• Evaluates cross-modal retrieval performance using metrics like P@1, R@1, and mAP across multiple datasets including ImageNet, AudioSet, UrbanSound8K, and ESC-50.

Experimental results

Research questions

• RQ1 Can a contrastive learning framework like CLIP be successfully extended to include audio as a third modality without compromising performance on vision and language?
• RQ2 To what extent does joint training of audio, text, and image encoders improve zero-shot generalization in environmental sound classification?
• RQ3 How does partial training (audio head only) compare to full end-to-end fine-tuning in terms of classification and cross-modal querying performance?
• RQ4 What is the impact of the AudioSet dataset on the model’s ability to generalize to downstream audio benchmarks like UrbanSound8K and ESC-50?
• RQ5 Can AudioCLIP perform effective cross-modal querying between text, images, and audio in both zero-shot and fine-tuned settings?

Key findings

• AudioCLIP achieves a new state-of-the-art accuracy of 90.07% on the UrbanSound8K dataset and 97.15% on the ESC-50 dataset under full training.
• The model sets a new zero-shot classification baseline of 68.78% on UrbanSound8K and 69.40% on ESC-50, significantly outperforming prior methods.
• Partial training of the audio head alone already achieves 89.95% accuracy on UrbanSound8K and 96.65% on ESC-50, demonstrating strong zero-shot generalization.
• Full training improves cross-modal querying performance on AudioSet, UrbanSound8K, and ESC-50, particularly in mAP for audio-by-text and image-by-audio queries.
• The model's performance on ImageNet retrieval tasks degrades slightly after full training, indicating potential domain shift between ImageNet and AudioSet.
• The proposed model enables effective cross-modal retrieval across all combinations of text, image, and audio, with mAP scores exceeding 70% on ESC-50 for audio-by-text and text-by-audio queries after full training.