[Paper Review] Listen to What You Want: Neural Network-based Universal Sound Selector

Tsubasa Ochiai, Marc Delcroix+4 more|arXiv (Cornell University)|Jun 10, 2020

Speech and Audio Processing19 references4 citations

TL;DR

This paper proposes Sound Selector, a deep neural network that directly extracts multiple desired acoustic events (AEs) from audio mixtures using a learned n-hot class representation, bypassing the need for source separation or classification. It achieves high-quality sound selection with consistent performance across unseen numbers of sources and target classes, demonstrating strong generalization and computational efficiency.

ABSTRACT

Being able to control the acoustic events (AEs) to which we want to listen would allow the development of more controllable hearable devices. This paper addresses the AE sound selection (or removal) problems, that we define as the extraction (or suppression) of all the sounds that belong to one or multiple desired AE classes. Although this problem could be addressed with a combination of source separation followed by AE classification, this is a sub-optimal way of solving the problem. Moreover, source separation usually requires knowing the maximum number of sources, which may not be practical when dealing with AEs. In this paper, we propose instead a universal sound selection neural network that enables to directly select AE sounds from a mixture given user-specified target AE classes. The proposed framework can be explicitly optimized to simultaneously select sounds from multiple desired AE classes, independently of the number of sources in the mixture. We experimentally show that the proposed method achieves promising AE sound selection performance and could be generalized to mixtures with a number of sources that are unseen during training.

Motivation & Objective

• To address the limitations of cascaded source separation and classification for selecting desired acoustic events (AEs) from complex audio mixtures.
• To develop a unified framework that directly extracts target AEs without requiring prior knowledge of the number of sources in the mixture.
• To enable simultaneous selection of multiple AE classes with a single forward pass, reducing computational cost compared to iterative methods.
• To improve generalization to unseen mixture conditions, such as higher numbers of AE classes or more target classes than seen during training.
• To provide a computationally efficient, end-to-end solution for AE sound selection suitable for real-time hearable devices.

Proposed method

• The Sound Selector uses a deep neural network (DNN) that takes the raw time-domain audio mixture and a one-hot or n-hot vector indicating target AE classes as inputs.
• The network directly estimates the time-domain waveform of the selected AE sounds using a differentiable loss function optimized for sound selection performance.
• For multi-class selection, the network is trained to simultaneously extract multiple AE classes in a single forward pass, using a shared architecture with joint optimization.
• The method avoids permutation ambiguity and fixed-source limitations inherent in permutation-invariant training (PIT) based separation by directly modeling selection rather than separation.
• The framework is trained end-to-end on a dataset of synthetic mixtures with varying numbers of AE classes and target combinations.
• The model generalizes to mixtures with more AE classes than seen during training, including 7-class mixtures and 4-class simultaneous selection, without retraining.

Experimental results

Research questions

• RQ1 Can a deep neural network directly extract multiple desired acoustic events from a mixture without prior source separation, using only class labels as guidance?
• RQ2 How does the performance of direct sound selection compare to cascaded approaches combining source separation and classification?
• RQ3 Can the proposed method generalize to mixtures with more AE classes than seen during training?
• RQ4 Does simultaneous multi-class selection outperform iterative single-class extraction in terms of quality and efficiency?
• RQ5 What is the generalization capability of the model to unseen numbers of sources and target classes in real-world-like audio scenes?

Key findings

• The proposed Sound Selector achieves an average SDR improvement of 6.9 dB for two-class selection across mixtures with 3 to 5 AE classes, outperforming iterative methods in some cases.
• For four-class simultaneous selection, the model achieves a mean SDR improvement of 5.3 dB on unseen 7-class mixtures, demonstrating strong generalization to higher complexity.
• The simultaneous extraction scheme achieves comparable or better performance than iterative extraction with significantly lower computational cost.
• Even in high-complexity settings with three target classes, the model maintains a high absolute SDR of 16.9 dB, indicating minimal signal distortion.
• The model generalizes effectively to mixtures with seven AE classes and four target classes, which were not included in the training data, confirming robustness to unseen conditions.
• Sound removal experiments show consistent SDR improvements of about 6 dB, confirming the method's effectiveness for both selection and suppression tasks.