[Paper Review] End-to-End Multi-Channel Speech Separation

Rongzhi Gu, Jian Wu+7 more|arXiv (Cornell University)|May 15, 2019

Speech and Audio Processing23 references80 citations

TL;DR

The paper proposes an end-to-end time-domain multi-channel speech separation model that learns spatial features directly from data by reformulating STFT/IPD as time-domain convolution kernels and making them learnable.

ABSTRACT

The end-to-end approach for single-channel speech separation has been studied recently and shown promising results. This paper extended the previous approach and proposed a new end-to-end model for multi-channel speech separation. The primary contributions of this work include 1) an integrated waveform-in waveform-out separation system in a single neural network architecture. 2) We reformulate the traditional short time Fourier transform (STFT) and inter-channel phase difference (IPD) as a function of time-domain convolution with a special kernel. 3) We further relaxed those fixed kernels to be learnable, so that the entire architecture becomes purely data-driven and can be trained from end-to-end. We demonstrate on the WSJ0 far-field speech separation task that, with the benefit of learnable spatial features, our proposed end-to-end multi-channel model significantly improved the performance of previous end-to-end single-channel method and traditional multi-channel methods.

Motivation & Objective

• Motivate end-to-end multi-channel speech separation to address reverberation and spatial cues in far-field scenes.
• Integrate waveform-in and waveform-out processing within a single neural network.
• Reformulate STFT and IPD as learnable time-domain convolutions to enable end-to-end training.
• Demonstrate improved performance over previous end-to-end single-channel methods and traditional multi-channel approaches on WSJ0 far-field data.

Proposed method

• Extend TasNet-style time-domain separation to multi-channel inputs.
• Incorporate frequency-domain spatial features (IPD) via cross-domain learning in early, middle, or late fusion schemes.
• Reformulate STFT and IPD as convolutions with learnable kernels, enabling end-to-end learning of IPD from time-domain signals.
• Train the entire system with SI-SNR loss in an end-to-end fashion.
• Evaluate several baselines and rival architectures, including cross-domain training, parallel encoders, and end-to-end separation with learnable kernels.

Experimental results

Research questions

• RQ1 Can an end-to-end time-domain multi-channel separation system outperform both traditional multi-channel spectral methods and single-channel end-to-end models on far-field data?
• RQ2 Does learnable time-domain IPD computation within a single network improve spatial separation performance?
• RQ3 What is the impact of different fusion strategies (early, middle, late) for integrating time-domain and frequency-domain spatial features?
• RQ4 How does the end-to-end model compare to cross-domain and cascaded approaches on WSJ0 2-mix reverberant data?

Key findings

• End-to-end multi-channel separation with trainable IPD kernels and cosIPD/sinIPD features achieves strong SI-SNR improvements on WSJ0 far-field data.
• Cross-domain training with LPS and IPD features improves performance over single-channel TasNet and some frequency-domain baselines.
• Fixed STFT-like kernels provide strong initial IPD representations, while learnable kernels (especially with sinIPD) yield the best performance among the tested configurations.
• The proposed end-to-end approach surpasses ideal time-frequency masks (IBM/IRM/IPSM) in the reported setup.
• Parallel encoder and fusion strategies further enhance performance in samples with large inter-channel angle differences.