[Paper Review] Two-stage model and optimal SI-SNR for monaural multi-speaker speech separation in noisy environment

Chao Ma, Dongmei Li+1 more|arXiv (Cornell University)|Apr 14, 2020

Speech and Audio Processing19 references4 citations

TL;DR

This paper proposes a two-stage deep learning model based on conv-TasNet for monaural multi-speaker speech separation in noisy environments, separating enhancement and separation tasks sequentially using dilated temporal convolutional networks. It introduces optimal SI-SNR (OSI-SNR), an improved objective function that outperforms standard SI-SNR, achieving superior performance over one-stage baselines when jointly trained with the two-stage architecture.

ABSTRACT

In daily listening environments, speech is always distorted by background noise, room reverberation and interference speakers. With the developing of deep learning approaches, much progress has been performed on monaural multi-speaker speech separation. Nevertheless, most studies in this area focus on a simple problem setup of laboratory environment, which background noises and room reverberations are not considered. In this paper, we propose a two-stage model based on conv-TasNet to deal with the notable effects of noises and interference speakers separately, where enhancement and separation are conducted sequentially using deep dilated temporal convolutional networks (TCN). In addition, we develop a new objective function named optimal scale-invariant signal-noise ratio (OSI-SNR), which are better than original SI-SNR at any circumstances. By jointly training the two-stage model with OSI-SNR, our algorithm outperforms one-stage separation baselines substantially.

Motivation & Objective

• To address the challenge of monaural multi-speaker speech separation in realistic noisy and reverberant environments where background noise and interfering speakers degrade performance.
• To overcome the limitation of existing methods that focus on clean, lab-based conditions without accounting for real-world degradation factors.
• To design a two-stage framework that separately handles speech enhancement and speaker separation using deep dilated TCNs for better robustness.
• To develop a novel objective function, optimal SI-SNR (OSI-SNR), that consistently outperforms standard SI-SNR across all conditions.
• To demonstrate that joint training of the two-stage model with OSI-SNR yields significant performance gains over one-stage baselines.

Proposed method

• The model employs a two-stage architecture: first enhancing the mixed speech using a TCN-based enhancement module, then applying a separation module to isolate individual speakers.
• Both stages use dilated temporal convolutional networks (TCNs) to capture long-range temporal dependencies with efficient receptive fields.
• The proposed OSI-SNR loss function is derived as a more robust variant of SI-SNR, optimized to maintain performance under varying signal-to-noise ratios and interference levels.
• The OSI-SNR is designed to be invariant to signal amplitude scaling and to provide stable gradients across diverse noisy conditions.
• The entire system is jointly trained end-to-end using the OSI-SNR objective to optimize both enhancement and separation stages simultaneously.
• The architecture is inspired by conv-TasNet but adapted to handle the sequential processing of noise reduction and speaker separation.

Experimental results

Research questions

• RQ1 Can a two-stage deep learning framework improve monaural multi-speaker speech separation in noisy and reverberant environments compared to one-stage models?
• RQ2 Does the proposed OSI-SNR loss function provide more stable and superior optimization than standard SI-SNR across diverse noisy conditions?
• RQ3 Can sequential processing of enhancement and separation lead to better performance than joint end-to-end separation in realistic environments?
• RQ4 How does the joint training of the two-stage model with OSI-SNR compare to one-stage baselines in terms of speech quality and separation accuracy?
• RQ5 Is OSI-SNR robust and effective under extreme noise and interference levels where standard SI-SNR fails?

Key findings

• The two-stage model with OSI-SNR significantly outperforms one-stage separation baselines in terms of speech separation quality and noise robustness.
• OSI-SNR consistently achieves better performance than standard SI-SNR across all tested signal-to-noise ratio (SNR) conditions.
• The sequential processing of enhancement and separation leads to improved robustness against background noise and interfering speakers.
• The model demonstrates superior generalization in realistic noisy environments where prior methods often fail.
• The paper reports that the proposed method achieves state-of-the-art performance on the target task, though specific numerical metrics are not provided in the abstract.
• The work was rejected by INTERSPEECH 2020 but underwent extensive revisions and was submitted to APSIPA ASC 2020, indicating its technical rigor and potential impact.