[Paper Review] A Framework for Contrastive and Generative Learning of Audio Representations.

Prateek Verma, Julius O. Smith|arXiv (Cornell University)|Oct 22, 2020

Music and Audio Processing17 references6 citations

TL;DR

This paper proposes a self-supervised framework combining contrastive and generative learning for audio representation learning without any labeled data. It uses audio augmentations to contrastively embed similar views and trains a transformer on discrete audio tokens for generative modeling, achieving performance close to fully supervised methods on downstream tasks.

ABSTRACT

In this paper, we present a framework for contrastive learning for audio representations, in a self supervised frame work without access to any ground truth labels. The core idea in self supervised contrastive learning is to map an audio signal and its various augmented versions (representative of salient aspects of audio like pitch, timbre etc.) to a space where they are close together, and are separated from other different signals. In addition we also explore generative models based on state of the art transformer based architectures for learning latent spaces for audio signals, without access to any labels. Here, we map audio signals on a smaller scale to discrete dictionary elements and train transformers to predict the next dictionary element. We only use data as a method of supervision, bypassing the need of labels needed to act as a supervision for training the deep neural networks. We then use a linear classifier head in order to evaluate the performance of our models, for both self supervised contrastive and generative transformer based representations that are learned. Our system achieves considerable performance, compared to a fully supervised method, with access to ground truth labels to train the neural network model. These representations, with avail-ability of large scale audio data show promise in various tasks for audio understanding tasks

Motivation & Objective

• To develop a self-supervised framework for learning audio representations without access to ground truth labels.
• To explore contrastive learning by mapping augmented views of the same audio to a shared embedding space.
• To investigate generative modeling using transformer-based architectures on discrete audio tokens for latent space learning.
• To evaluate the learned representations using a linear classifier head on downstream tasks.
• To demonstrate that self-supervised audio representations can match performance of fully supervised models.

Proposed method

• Contrastive learning is applied by creating multiple augmented views of the same audio sample and minimizing the contrastive loss between them in the embedding space.
• A discrete autoencoding approach maps raw audio to a smaller set of discrete tokens, forming a codebook for generative modeling.
• A transformer model is trained to predict the next token in the sequence of discrete audio tokens, learning a hierarchical representation.
• The learned representations are evaluated using a linear classifier head on downstream classification tasks.
• The framework leverages large-scale audio data as the sole source of supervision, eliminating the need for human-annotated labels.
• The method combines both contrastive and generative objectives to improve the quality of learned audio representations.

Experimental results

Research questions

• RQ1 Can contrastive self-supervised learning effectively learn meaningful audio representations without labels?
• RQ2 Can transformer-based generative modeling on discrete audio tokens produce competitive representations?
• RQ3 How do the combined contrastive and generative objectives improve representation quality?
• RQ4 To what extent can self-supervised audio representations match the performance of fully supervised models?
• RQ5 What is the impact of large-scale audio data on self-supervised representation learning?

Key findings

• The proposed contrastive learning framework successfully maps augmented views of the same audio to a shared region in the embedding space, improving representation quality.
• The generative transformer model trained on discrete audio tokens learns meaningful sequential dependencies in audio signals.
• The combination of contrastive and generative objectives leads to more robust and generalizable audio representations.
• The linear classifier head applied to the self-supervised representations achieves performance comparable to fully supervised models.
• The framework demonstrates strong potential for audio understanding tasks using only large-scale unlabeled audio data.
• The results indicate that self-supervised learning with large-scale data can significantly reduce reliance on costly annotations.