[Paper Review] ESPnet-se: end-to-end speech enhancement and separation toolkit designed for asr integration

Chenda Li, Jing Shi+9 more|arXiv (Cornell University)|Nov 7, 2020

Speech and Audio ProcessingComputer Science60 references75 citations

TL;DR

ESPnet-SE is an end-to-end speech enhancement and separation toolkit designed for seamless integration with automatic speech recognition (ASR), enabling joint training and evaluation of speech enhancement systems with ASR models. It supports single- and multi-channel, anechoic and reverberant, single- and multi-speaker scenarios, and achieves state-of-the-art performance, including 16.1% WER on WSJ0-2mix with joint optimization of separation and ASR.

ABSTRACT

We present ESPnet-SE, which is designed for the quick development of speech enhancement and speech separation systems in a single framework, along with the optional downstream speech recognition module. ESPnet-SE is a new project which integrates rich automatic speech recognition related models, resources and systems to support and validate the proposed front-end implementation (i.e. speech enhancement and separation).It is capable of processing both single-channel and multi-channel data, with various functionalities including dereverberation, denoising and source separation. We provide all-in-one recipes including data pre-processing, feature extraction, training and evaluation pipelines for a wide range of benchmark datasets. This paper describes the design of the toolkit, several important functionalities, especially the speech recognition integration, which differentiates ESPnet-SE from other open source toolkits, and experimental results with major benchmark datasets.

Motivation & Objective

• To develop a unified, open-source toolkit that integrates end-to-end speech enhancement and separation (E2E-SE) with automatic speech recognition (ASR) for joint optimization.
• To enable seamless training and evaluation of E2E-SE models using ASR loss, even in the absence of clean reference speech signals, by leveraging transcription data.
• To support diverse input conditions including single/multi-channel, anechoic/reverberant, and single/multi-speaker scenarios within a single framework.
• To provide out-of-the-box, all-in-one recipes for data preparation, feature extraction, training, and evaluation on benchmark datasets.
• To differentiate from existing toolkits by enabling direct integration with ESPnet’s rich ASR models for joint optimization and performance evaluation.

Proposed method

• Integrates state-of-the-art E2E-SE models—TasNet, DPRNN, T-F masking, and neural beamformer—into a unified PyTorch-based framework.
• Supports raw waveform input and applies differentiable STFT for frequency-domain models, enabling end-to-end training with backpropagation.
• Provides unified, stage-by-stage recipes for data preprocessing, feature extraction, model training, and evaluation across multiple benchmark datasets.
• Enables optional integration with ESPnet’s ASR models (e.g., Transformer, CTC, attention-based encoder-decoder) for joint training using ASR loss.
• Supports joint optimization using both E2E-SE loss and ASR loss, allowing direct training on weakly supervised data (e.g., CHiME-4) without clean references.
• Uses pretrained ASR models (e.g., on WSJ) to evaluate enhanced/separated speech, enabling zero-shot evaluation of E2E-SE performance via WER/CER.

Experimental results

Research questions

• RQ1 Can end-to-end speech enhancement and separation models be effectively integrated with automatic speech recognition (ASR) in a unified framework to improve robustness?
• RQ2 How does joint optimization of E2E-SE and ASR modules affect recognition performance on noisy or reverberant speech?
• RQ3 What is the performance of E2E-SE models on benchmark datasets like WSJ0-2mix and spatialized WSJ0-2mix when evaluated with ASR metrics?
• RQ4 Can E2E-SE systems achieve competitive performance without access to clean reference speech signals by leveraging only transcriptions?
• RQ5 How does the integration of ASR into the E2E-SE pipeline compare to standalone E2E-SE or standalone ASR in terms of robustness and generalization?

Key findings

• The integration of Conv-TasNet with an ASR model achieves 16.7% WER on the WSJ0-2mix evaluation set, demonstrating strong end-to-end performance.
• The DPRNN-based E2E-SE model achieves 16.1% WER on WSJ0-2mix when jointly optimized with ASR, outperforming standalone ASR on WSJ in some cases.
• On the spatialized WSJ0-2mix dataset, the neural beamformer model achieves over 7 dB SDR improvement in anechoic conditions compared to prior work.
• In reverberant conditions, the neural beamformer achieves performance on par with prior work, with further gains when combined with a WPE dereverberation module.
• Joint training of E2E-SE and ASR leads to improved ASR performance (e.g., 15.8% WER with Conv-TasNet + Transformer) even when trained only on the WSJ0-2mix dataset.
• Visualization of spectrograms confirms that the DPRNN model effectively separates source signals from challenging same-gender mixtures, with high SDR (18.54 dB and 12.88 dB on two test samples).