[Paper Review] Learning from Between-class Examples for Deep Sound Recognition

Yuji Tokozume, Yoshitaka Ushiku+1 more|arXiv (Cornell University)|Nov 28, 2017

Music and Audio Processing14 references153 citations

TL;DR

BC learning mixes two sounds from different classes and trains the model to predict the mixing ratio, yielding improved accuracy across networks and datasets and surpassing human level on ESC-50 with EnvNet-v2.

ABSTRACT

Deep learning methods have achieved high performance in sound recognition tasks. Deciding how to feed the training data is important for further performance improvement. We propose a novel learning method for deep sound recognition: Between-Class learning (BC learning). Our strategy is to learn a discriminative feature space by recognizing the between-class sounds as between-class sounds. We generate between-class sounds by mixing two sounds belonging to different classes with a random ratio. We then input the mixed sound to the model and train the model to output the mixing ratio. The advantages of BC learning are not limited only to the increase in variation of the training data; BC learning leads to an enlargement of Fisher's criterion in the feature space and a regularization of the positional relationship among the feature distributions of the classes. The experimental results show that BC learning improves the performance on various sound recognition networks, datasets, and data augmentation schemes, in which BC learning proves to be always beneficial. Furthermore, we construct a new deep sound recognition network (EnvNet-v2) and train it with BC learning. As a result, we achieved a performance surpasses the human level.

Motivation & Objective

• Motivate improved data utilization for deep sound recognition.
• Introduce Between-Class (BC) learning by mixing sounds from different classes.
• Train models to predict the mixing ratio to enlarge Fisher’s criterion.
• Demonstrate BC learning across multiple architectures and datasets.
• Show that BC learning can outperform human performance on ESC-50 with a deeper network.

Proposed method

• Create training samples by mixing two sounds from different classes with a random ratio.
• Use a mixing formula that accounts for sound pressure levels and compute a corresponding p to preserve perceived ratios (Eq. 2).
• Represent mixed-labels as t = r t1 + (1 - r) t2 and optimize using KL-divergence loss.
• Train with mini-batch SGD; BC learning may require more epochs than standard learning.
• Visualize feature space to argue enlargement of Fisher’s criterion and regularization of class relationships.

Experimental results

Research questions

• RQ1 Does BC learning improve recognition performance across architectures, datasets, and data augmentation schemes?
• RQ2 How should two sounds be mixed (and how should labels be assigned) to maximize BC effectiveness?
• RQ3 What is the effect of BC learning on Fisher’s criterion and class-relationship regularization in the feature space?
• RQ4 Can BC learning surpass human performance on challenging environmental sound datasets?

Key findings

• BC learning improves all evaluated networks (EnvNet, SoundNet5, M18, Logmel-CNN+BN, EnvNet-v2) across ESC-50, ESC-10, UrbanSound8K.
• On ESC-50 with EnvNet-v2, BC learning achieves 18.2% error (vs. 25.6% standard), with further gains under strong augmentation to 15.1%.
• BC learning yields larger Fisher’s criterion and regularizes class distributions, reducing misclassification of mixed-class sounds.
• EnvNet-v2 with BC learning surpasses human performance on ESC-50 (18.2% vs. 18.7% reported for humans in prior work).
• Ablation shows the proposed mixing method (Eq. 2 with A-weighting) and ratio labeling provide best performance (24.1% error on ESC-50).