[Paper Review] Unsupervised Sound Separation Using Mixture Invariant Training

Scott Wisdom, Efthymios Tzinis+4 more|arXiv (Cornell University)|Jun 23, 2020

Speech and Audio ProcessingComputer Science48 references91 citations

TL;DR

The paper introduces MixIT, a fully unsupervised framework for single-channel sound separation that learns from mixtures of mixtures and can be combined with supervised data for semi-supervised domain adaptation. MixIT achieves competitive performance with supervised methods on speech tasks and enables training from large amounts of unlabeled in-the-wild audio.

ABSTRACT

In recent years, rapid progress has been made on the problem of single-channel sound separation using supervised training of deep neural networks. In such supervised approaches, a model is trained to predict the component sources from synthetic mixtures created by adding up isolated ground-truth sources. Reliance on this synthetic training data is problematic because good performance depends upon the degree of match between the training data and real-world audio, especially in terms of the acoustic conditions and distribution of sources. The acoustic properties can be challenging to accurately simulate, and the distribution of sound types may be hard to replicate. In this paper, we propose a completely unsupervised method, mixture invariant training (MixIT), that requires only single-channel acoustic mixtures. In MixIT, training examples are constructed by mixing together existing mixtures, and the model separates them into a variable number of latent sources, such that the separated sources can be remixed to approximate the original mixtures. We show that MixIT can achieve competitive performance compared to supervised methods on speech separation. Using MixIT in a semi-supervised learning setting enables unsupervised domain adaptation and learning from large amounts of real world data without ground-truth source waveforms. In particular, we significantly improve reverberant speech separation performance by incorporating reverberant mixtures, train a speech enhancement system from noisy mixtures, and improve universal sound separation by incorporating a large amount of in-the-wild data.

Motivation & Objective

• Motivate unsupervised learning for single-channel sound separation to overcome reliance on ground-truth sources in synthetic training data.
• Introduce MixIT, a mixture-invariant training framework that uses mixtures of mixtures as references.
• Demonstrate MixIT on speech separation, speech enhancement, and universal sound separation tasks.
• Show benefits of semi-supervised training combining MixIT with supervised losses for domain adaptation.

Proposed method

• Generalize permutation invariant training (PIT) to operate on unsupervised mixtures by introducing MixIT, which separates a mixture of mixtures into a larger set of latent sources.
• Construct MoMs (mixtures of mixtures) by summing two or more unlabeled mixtures and train a separation model to remap the estimated sources to the original mixtures via a mixing matrix A with binary columns.
• Use a signal-level loss (negative SNR with a soft threshold) and perform an exhaustive binary search over A to minimize the remixed reconstruction loss.
• Optionally combine MixIT with supervised PIT in a semi-supervised setting, mixing p% supervised data with MixIT-unsupervised data in the same training batch.
• Adopt a time-domain convolutional network (TD-CN++ style) with a mixture-consistency projection layer to ensure the outputs sum to the input mixture.

Experimental results

Research questions

• RQ1 Can a fully unsupervised training objective base separation performance on single-channel mixtures without ground-truth references?
• RQ2 How does MixIT enable domain adaptation and semi-supervised learning when target-domain mixtures lack ground-truth sources?
• RQ3 What is the impact of MixIT on speech separation, speech enhancement, and universal sound separation tasks?
• RQ4 Can combining MixIT with limited supervised data approach or match state-of-the-art supervised methods in practice?

Key findings

• MixIT enables completely unsupervised single-channel separation using mixtures of mixtures as references, achieving competitive performance to supervised methods on speech separation tasks.
• Semi-supervised training that mixes supervised and MixIT losses improves performance, especially for domain-mismatched data, enabling better cross-domain adaptation.
• Across speech separation, enhancement, and universal sound separation, MixIT shows strong benefits in domain adaptation and can leverage large amounts of unlabeled real-world data.
• In speech enhancement, a MixIT-based model trained only with mixture labels achieves substantial SI-SNRi, approaching 76% of a fully supervised model on a proxy task.
• For universal sound separation, MixIT with in-the-wild data yields meaningful improvements over purely supervised setups in several configurations.