[Paper Review] Enhancing End-to-End Multi-channel Speech Separation via Spatial Feature Learning

Rongzhi Gu, Shixiong Zhang+6 more|arXiv (Cornell University)|Mar 9, 2020

Speech and Audio Processing17 references4 citations

TL;DR

This paper proposes an end-to-end multi-channel speech separation model that learns spatial features directly from raw waveforms using a 2D convolutional layer to compute inter-channel convolution differences (ICDs), a data-driven alternative to hand-crafted inter-channel phase differences (IPDs). The ICD-based model improves SI-SDRi by 10.4% over IPD-based models on a simulated WSJ0 2-mix dataset, demonstrating superior performance through end-to-end optimization of spatial cues.

ABSTRACT

Hand-crafted spatial features (e.g., inter-channel phase difference, IPD) play a fundamental role in recent deep learning based multi-channel speech separation (MCSS) methods. However, these manually designed spatial features are hard to incorporate into the end-to-end optimized MCSS framework. In this work, we propose an integrated architecture for learning spatial features directly from the multi-channel speech waveforms within an end-to-end speech separation framework. In this architecture, time-domain filters spanning signal channels are trained to perform adaptive spatial filtering. These filters are implemented by a 2d convolution (conv2d) layer and their parameters are optimized using a speech separation objective function in a purely data-driven fashion. Furthermore, inspired by the IPD formulation, we design a conv2d kernel to compute the inter-channel convolution differences (ICDs), which are expected to provide the spatial cues that help to distinguish the directional sources. Evaluation results on simulated multi-channel reverberant WSJ0 2-mix dataset demonstrate that our proposed ICD based MCSS model improves the overall signal-to-distortion ratio by 10.4% over the IPD based MCSS model.

Motivation & Objective

• To address the data mismatch between hand-crafted spatial features (e.g., IPD) and data-driven end-to-end speech separation frameworks.
• To develop a fully end-to-end MCSS system that learns spatial cues directly from raw waveforms without relying on fixed, precomputed features.
• To overcome the limitations of phase reconstruction and reverberation effects in far-field speech separation by integrating learned spatial representations.
• To design a novel spatial feature learning mechanism—inter-channel convolution differences (ICDs)—that is optimized jointly with the separation objective.

Proposed method

• A 2D convolutional layer is used to learn time-domain spatial filters across multiple microphone channels, enabling adaptive spatial filtering from raw waveforms.
• A specialized conv2d kernel is designed to compute inter-channel convolution differences (ICDs), inspired by the mathematical formulation of IPD but learned end-to-end.
• The encoder-decoder structure replaces STFT and iSTFT with learnable filters, maintaining temporal alignment and enabling end-to-end training.
• The ICDs are computed using a window function $ w $, initialized to -1 for exact difference computation, and made learnable to adapt to optimal spatial representations.
• The model is trained using scale-invariant signal-to-distortion ratio (SI-SDR) as the objective, with permutation-invariant training to handle speaker label ambiguity.
• The architecture integrates ICDs with the encoder output and separation module, allowing the network to jointly optimize for speech separation and spatial feature learning.

Experimental results

Research questions

• RQ1 Can learned spatial features, computed directly from raw waveforms via a 2D convolutional layer, outperform hand-crafted inter-channel phase differences (IPDs) in end-to-end multi-channel speech separation?
• RQ2 Does the proposed inter-channel convolution difference (ICD) mechanism provide more effective spatial cues than IPD when optimized end-to-end with a speech separation objective?
• RQ3 How does the initialization and learnability of the ICD kernel affect separation performance?
• RQ4 What is the contribution of ICDs in low-frequency regions, and do they complement IPD-based features?
• RQ5 Can the integration of ICDs and IPDs further improve separation performance, especially in challenging angular separation scenarios?

Key findings

• The ICD-based MCSS model achieves a 10.4% relative improvement in SI-SDRi over the IPD-based model on the simulated WSJ0 2-mix dataset.
• The model with 33 filters and learnable ICD kernel initialization achieves 12.3 dB SI-SDRi, outperforming the baseline SC-Conv-TasNet (9.1 dB) and the IPD-based model (11.5 dB).
• The ICD-based model shows a 0.4 dB improvement over the IPD-based model, indicating that data-driven spatial features can be more effective than hand-crafted ones.
• The performance of ICD-based models is significantly worse under small angle differences (<15°), likely due to insufficient training data in that range.
• Combining ICDs with cosIPD and sinIPD yields the best performance (12.4 dB SI-SDRi), suggesting complementary spatial information from both features.
• Visualization of learned filters shows that ICDs primarily respond to low-frequency components, indicating they capture complementary spatial cues to IPDs.