[Paper Review] Zero-Shot Audio Classification via Semantic Embeddings

Huang Xie, Tuomas Virtanen|arXiv (Cornell University)|Nov 24, 2020

Music and Audio Processing37 references4 citations

TL;DR

This paper proposes a zero-shot audio classification method that leverages semantic embeddings from textual labels or sentence descriptions to classify audio clips from unseen sound classes. By using a bilinear compatibility function between VGGish-based acoustic embeddings and semantic embeddings (e.g., Word2Vec, GloVe, BERT), the model achieves significant performance gains on ESC-50 and AudioSet, especially when combining multiple language model outputs via concatenation.

ABSTRACT

In this paper, we study zero-shot learning in audio classification via semantic embeddings extracted from textual labels and sentence descriptions of sound classes. Our goal is to obtain a classifier that is capable of recognizing audio instances of sound classes that have no available training samples, but only semantic side information. We employ a bilinear compatibility framework to learn an acoustic-semantic projection between intermediate-level representations of audio instances and sound classes, i.e., acoustic embeddings and semantic embeddings. We use VGGish to extract deep acoustic embeddings from audio clips, and pre-trained language models (Word2Vec, GloVe, BERT) to generate either label embeddings from textual labels or sentence embeddings from sentence descriptions of sound classes. Audio classification is performed by a linear compatibility function that measures how compatible an acoustic embedding and a semantic embedding are. We evaluate the proposed method on a small balanced dataset ESC-50 and a large-scale unbalanced audio subset of AudioSet. The experimental results show that classification performance is significantly improved by involving sound classes that are semantically close to the test classes in training. Meanwhile, we demonstrate that both label embeddings and sentence embeddings are useful for zero-shot learning. Classification performance is improved by concatenating label/sentence embeddings generated with different language models. With their hybrid concatenations, the results are improved further.

Motivation & Objective

• To enable audio classification for sound classes without any training samples, using only semantic side information.
• To investigate the effectiveness of label embeddings and sentence embeddings in zero-shot audio classification.
• To improve classification performance by combining semantic embeddings from multiple pre-trained language models.
• To evaluate the method on both small balanced (ESC-50) and large unbalanced (AudioSet) datasets.
• To explore the impact of different embedding fusion strategies on zero-shot learning performance.

Proposed method

• Uses VGGish to extract deep acoustic embeddings from audio clips.
• Generates semantic embeddings using pre-trained language models (Word2Vec, GloVe, BERT) from textual labels or sentence descriptions of sound classes.
• Employs a bilinear compatibility function to measure compatibility between acoustic and semantic embeddings for classification.
• Concatenates label/sentence embeddings from different language models to form hybrid semantic representations.
• Trains a linear classifier using the compatibility scores between acoustic and semantic embeddings.
• Applies McNemar’s test to evaluate statistical significance of performance differences between embedding combinations.

Experimental results

Research questions

• RQ1 Can semantic embeddings derived from textual labels or sentence descriptions effectively enable zero-shot audio classification?
• RQ2 How does the performance of zero-shot audio classification vary when using label embeddings versus sentence embeddings?
• RQ3 Does concatenating semantic embeddings from multiple language models improve classification performance?
• RQ4 How does the choice of language model (e.g., Word2Vec, GloVe, BERT) affect the quality of semantic representations in zero-shot learning?
• RQ5 What is the impact of embedding fusion strategies (e.g., hybrid concatenation of label and sentence embeddings) on classification accuracy?

Key findings

• Classification performance on ESC-50 significantly improves when training includes sound classes semantically similar to the test classes.
• Both label embeddings and sentence embeddings are effective for zero-shot audio classification, with sentence embeddings showing stronger performance in some settings.
• Concatenating embeddings from multiple language models (e.g., Word2Vec, GloVe, BERT) leads to improved classification accuracy compared to individual models.
• Hybrid concatenation of label and sentence embeddings (e.g., WLE+BSE, GLE+BSE) yields the best results, with Top-1 accuracy of 48.5% on AudioSet.
• Statistical significance testing (McNemar’s test, p = 5.41e-13) confirms that the performance gain from WLE+GLE+BSE over GLE+BSE is highly significant.
• The method demonstrates strong generalization on large-scale, unbalanced datasets like AudioSet, indicating scalability beyond small benchmarks.