[Paper Review] RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing

Efthymios Tzinis, Yossi Adi+4 more|arXiv (Cornell University)|Feb 17, 2022

Speech Recognition and SynthesisComputer Science71 references45 citations

TL;DR

RemixIT is a self-supervised, continual self-training framework for speech enhancement that eliminates reliance on in-domain clean speech or noise data by generating bootstrapped training mixtures through permutation and remixing of pseudo-estimated speech and noise from a pre-trained teacher model. It achieves state-of-the-art performance on multiple datasets, including 7.3 dB SI-SDRi on DNS and 6.9 dB on WHAM! without any in-domain clean targets, demonstrating robust zero-shot domain adaptation and continual improvement via teacher-student distillation with dynamic pseudo-target refinement.

ABSTRACT

We present RemixIT, a simple yet effective self-supervised method for training speech enhancement without the need of a single isolated in-domain speech nor a noise waveform. Our approach overcomes limitations of previous methods which make them dependent on clean in-domain target signals and thus, sensitive to any domain mismatch between train and test samples. RemixIT is based on a continuous self-training scheme in which a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures. Then, by permuting the estimated clean and noise signals and remixing them together, we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network. Vice-versa, the teacher periodically refines its estimates using the updated parameters of the latest student models. Experimental results on multiple speech enhancement datasets and tasks not only show the superiority of our method over prior approaches but also showcase that RemixIT can be combined with any separation model as well as be applied towards any semi-supervised and unsupervised domain adaptation task. Our analysis, paired with empirical evidence, sheds light on the inside functioning of our self-training scheme wherein the student model keeps obtaining better performance while observing severely degraded pseudo-targets.

Motivation & Objective

• To address the limitation of existing speech enhancement methods that require in-domain clean speech or noise data for training.
• To develop a self-supervised learning framework that enables continual, data-efficient adaptation to in-domain noisy mixtures without access to ground-truth clean signals.
• To overcome domain shift issues in zero-shot and semi-supervised settings by leveraging pseudo-labels from a pre-trained out-of-domain teacher model.
• To enable robust, scalable domain adaptation using only in-domain noisy mixtures and OOD noise sources, avoiding reliance on external modality or in-domain data.

Proposed method

• A pre-trained teacher model on out-of-domain (OOD) data performs inference on in-domain noisy mixtures to generate pseudo-estimated speech and noise components.
• The estimated speech and noise components are randomly permuted across the batch and remixxed to form new bootstrapped mixtures, which serve as augmented training data for the student model.
• The student model is trained to regress toward the teacher’s pseudo-estimated clean speech using a signal-level loss function (e.g., SI-SDR), enabling self-supervised learning.
• The teacher model is periodically updated using the student’s weights via a running mean or sequential update protocol, enabling continual refinement of pseudo-labels.
• An extension allows mixing in-domain noise recordings with teacher estimates via a Bernoulli sampling strategy to further improve performance.
• The framework supports both unsupervised and semi-supervised learning and is compatible with any separation model architecture.

Experimental results

Research questions

• RQ1 Can a self-training framework achieve state-of-the-art speech enhancement performance without requiring any in-domain clean speech or noise waveforms?
• RQ2 How does continual, bi-directional knowledge distillation between teacher and student models improve performance when pseudo-targets are severely degraded?
• RQ3 To what extent can bootstrapped remixing of teacher-estimated components generalize across domain shifts in real-world noisy speech?
• RQ4 Can the method achieve effective zero-shot and semi-supervised domain adaptation using only in-domain noisy mixtures and OOD data?
• RQ5 How does the integration of isolated in-domain noise recordings affect performance in the bootstrapped remixing process?

Key findings

• RemixIT achieves a SI-SDRi of 7.3 dB on the DNS test set without any in-domain clean data, outperforming the initial OOD teacher (6.1 dB) and state-of-the-art baselines.
• On the WHAM! dataset, RemixIT achieves 6.9 dB SI-SDRi in a semi-supervised setup using only in-domain mixtures and OOD teacher estimates, surpassing prior self-supervised methods.
• The method demonstrates consistent performance gains across multiple datasets, including DNS, LFSD, and WHAM!, even when starting from a weak OOD teacher.
• Incorporating isolated in-domain noise recordings into the remixing process further improves performance, with SI-SDRi increasing from 6.1 dB to 6.4 dB on DNS and 8.6 dB to 9.0 dB on WHAM! when using equal amounts of in-domain noise and mixture data.
• Empirical and theoretical analysis confirms that the student model continues to improve even when trained on extremely noisy pseudo-targets, validating the robustness of the self-training dynamics.
• The framework enables effective zero-shot and semi-supervised domain adaptation, with performance comparable to supervised baselines when limited in-domain data is available.