[Paper Review] Exploring modality-agnostic representations for music classification

Ho-Hsiang Wu, Magdalena Fuentes+1 more|arXiv (Cornell University)|Jun 2, 2021

Music and Audio Processing31 references4 citations

TL;DR

This paper proposes a modality-agnostic music classification framework that leverages cross-modal retrieval as a self-supervised pretext task to learn joint audio and image embeddings. By training a translation model between pre-trained audio and image embeddings, the method enables a single classifier to perform music instrument classification using either modality, achieving ~70% of the best single-modality system's performance in zero-shot settings with limited labeled data.

ABSTRACT

Music information is often conveyed or recorded across multiple data modalities including but not limited to audio, images, text and scores. However, music information retrieval research has almost exclusively focused on single modality recognition, requiring development of separate models for each modality. Some multi-modal works require multiple coexisting modalities given to the model as inputs, constraining the use of these models to the few cases where data from all modalities are available. To the best of our knowledge, no existing model has the ability to take inputs from varying modalities, e.g. images or sounds, and classify them into unified music categories. We explore the use of cross-modal retrieval as a pretext task to learn modality-agnostic representations, which can then be used as inputs to classifiers that are independent of modality. We select instrument classification as an example task for our study as both visual and audio components provide relevant semantic information. We train music instrument classifiers that can take both images or sounds as input, and perform comparably to sound-only or image-only classifiers. Furthermore, we explore the case when there is limited labeled data for a given modality, and the impact in performance by using labeled data from other modalities. We are able to achieve almost 70% of best performing system in a zero-shot setting. We provide a detailed analysis of experimental results to understand the potential and limitations of the approach, and discuss future steps towards modality-agnostic classifiers.

Motivation & Objective

• To develop a music classification system that works across multiple modalities (audio and image) without requiring all modalities at inference time.
• To investigate whether self-supervised cross-modal learning can produce modality-agnostic representations suitable for downstream classification.
• To evaluate performance under low-data regimes, especially when one modality has limited labeled examples.
• To compare the effectiveness of learned translation models versus simple dimensionality reduction (e.g., PCA) for joint embedding learning.
• To analyze the generalization capability of the model in zero-shot settings where the target modality is unseen during training.

Proposed method

• Pre-trained audio embeddings are obtained from a convolutional neural network trained on AudioSet, and image embeddings from a ResNet-50 model fine-tuned on ImageNet.
• A cross-modal translation model is trained in a self-supervised manner to map audio and image embeddings into a shared latent space using contrastive learning.
• The joint embedding space is optimized using a contrastive loss that encourages positive pairs (same instrument) to be close and negative pairs to be far apart.
• Principal component analysis (PCA) is used as an alternative to the translation model for dimensionality reduction and alignment of embeddings.
• A downstream classifier is trained on the joint embeddings using labeled data from either modality, enabling modality-agnostic inference.
• Performance is evaluated via cross-modal retrieval and instrument classification accuracy under various data availability settings.

Experimental results

Research questions

• RQ1 Can self-supervised cross-modal retrieval learning produce joint representations that enable effective modality-agnostic music classification?
• RQ2 How does the performance of a modality-agnostic classifier compare to single-modality classifiers when trained on limited data from one modality?
• RQ3 To what extent can a model generalize to a modality it was not explicitly trained on, in a zero-shot setting?
• RQ4 Does learning a translation model between modalities outperform simple dimensionality reduction (e.g., PCA) for joint representation learning?
• RQ5 How does the availability of labeled data from one modality affect the performance of a classifier trained on joint embeddings?

Key findings

• The modality-agnostic classifier achieves ~70% of the performance of the best single-modality system in a zero-shot setting, demonstrating strong generalization across modalities.
• The use of a learned translation model between audio and image embeddings outperforms PCA-based alignment in both cross-modal retrieval and classification tasks.
• When labeled data is scarce in one modality, performance can be significantly improved by leveraging labeled data from the other modality through joint representation learning.
• The model maintains high performance even when only one modality is available at inference time, confirming its modality-agnostic nature.
• Cross-modal retrieval performance correlates strongly with downstream classification accuracy, validating the effectiveness of the joint embedding space.
• The approach enables effective transfer learning from high-resource modalities (e.g., image) to low-resource ones (e.g., audio), especially in low-data regimes.