[Paper Review] LibriMix: An Open-Source Dataset for Generalizable Speech Separation

Joris Cosentino, Manuel Pariente+3 more|arXiv (Cornell University)|May 22, 2020

Speech and Audio ProcessingComputer Science31 references183 citations

TL;DR

LibriMix provides open-source two- and three-speaker mixtures (with and without noise) derived from LibriSpeech to improve generalization in speech separation, including cross-dataset evaluation and sparsely overlapping test sets.

ABSTRACT

In recent years, wsj0-2mix has become the reference dataset for single-channel speech separation. Most deep learning-based speech separation models today are benchmarked on it. However, recent studies have shown important performance drops when models trained on wsj0-2mix are evaluated on other, similar datasets. To address this generalization issue, we created LibriMix, an open-source alternative to wsj0-2mix, and to its noisy extension, WHAM!. Based on LibriSpeech, LibriMix consists of two- or three-speaker mixtures combined with ambient noise samples from WHAM!. Using Conv-TasNet, we achieve competitive performance on all LibriMix versions. In order to fairly evaluate across datasets, we introduce a third test set based on VCTK for speech and WHAM! for noise. Our experiments show that the generalization error is smaller for models trained with LibriMix than with WHAM!, in both clean and noisy conditions. Aiming towards evaluation in more realistic, conversation-like scenarios, we also release a sparsely overlapping version of LibriMix's test set.

Motivation & Objective

• Address generalization gaps when models are trained on WSJ0-2mix by providing a more diverse open-source dataset.
• Create LibriMix (clean and noisy, two- and three-speaker mixtures) derived from LibriSpeech with WHAM! noises for realistic evaluation.
• Enable fair cross-dataset evaluation with an added VCTK-based test set and sparsely overlapping test sets to mimic real-world conversations.
• Provide open-source generation scripts and demonstrate generalization improvements using Conv-TasNet.

Proposed method

• Construct LibriMix datasets from LibriSpeech and WHAM! noises to produce two- and three-speaker mixtures in clean and noisy conditions.
• Use LUFS-based loudness normalization for mixture scaling instead of simple signal-power-based SNRs.
• Mix utterances from multiple speakers randomly to generate training, development, and test sets, with train sets having a wide speaker base.
• Evaluate Conv-TasNet (via Asteroid implementation) on LibriMix and compare to ideal masks (IRM/IBM) under different sampling rates and modes (min/max).
• Introduce SparseLibriMix variants with varying overlap (0–100%) to study performance on sparsely overlapping mixtures.

Experimental results

Research questions

• RQ1 Does LibriMix improve generalization of deep speech separation models to other datasets (e.g., VCTK-2mix) compared to WHAM! and wsj0-2mix?
• RQ2 How do clean versus noisy conditions affect generalization when trained on LibriMix versus WHAM!?
• RQ3 What is the impact of mixture overlap and sparsity on separation performance for two- and three-speaker mixtures?
• RQ4 Can LibriMix enable reliable cross-dataset evaluation and support three-speaker noisy separation?

Key findings

• Conv-TasNet achieve competitive SI-SDR improvements on LibriMix, outperforming or matching ideal masks in several clean/noisy two-speaker cases.
• Models trained on LibriMix generalize better to VCTK-2mix than models trained on WHAM!, indicating improved cross-dataset generalization.
• LibriMix requires more data (train-360) than train-100 for better generalization, highlighting data quantity as a key factor.
• Three-speaker mixtures remain challenging, with oracle (IRM/IBM) still outperforming Conv-TasNet in many cases, especially in noisy settings.
• SparseLibriMix results show that higher overlap reduces SI-SDRi, and that Conv-TasNet can be outperformed by IRM in low-overlap regimes, suggesting room for improvement in sparsely overlapping scenarios.
• Libri3Mix is the first open-source dataset enabling three-speaker noisy separation, expanding evaluation beyond two-speaker cases.