[Paper Review] A Unified Framework for Speech Separation

Fahimeh Bahmaninezhad, Shixiong Zhang+4 more|arXiv (Cornell University)|Dec 17, 2019

Speech and Audio Processing42 references4 citations

TL;DR

This paper proposes a unified deep learning framework for speech separation that integrates both spectrogram-based and waveform-based approaches within a single architecture, differing only in kernel functions for encoding/decoding. The framework enables end-to-end training for single- and multi-channel setups, achieving state-of-the-art performance with improved stability, memory efficiency, and flexibility in latency control, particularly excelling in spectrogram-based separation for both single- and multi-channel settings.

ABSTRACT

Speech separation refers to extracting each individual speech source in a given mixed signal. Recent advancements in speech separation and ongoing research in this area, have made these approaches as promising techniques for pre-processing of naturalistic audio streams. After incorporating deep learning techniques into speech separation, performance on these systems is improving faster. The initial solutions introduced for deep learning based speech separation analyzed the speech signals into time-frequency domain with STFT; and then encoded mixed signals were fed into a deep neural network based separator. Most recently, new methods are introduced to separate waveform of the mixed signal directly without analyzing them using STFT. Here, we introduce a unified framework to include both spectrogram and waveform separations into a single structure, while being only different in the kernel function used to encode and decode the data; where, both can achieve competitive performance. This new framework provides flexibility; in addition, depending on the characteristics of the data, or limitations of the memory and latency can set the hyper-parameters to flow in a pipeline of the framework which fits the task properly. We extend single-channel speech separation into multi-channel framework with end-to-end training of the network while optimizing the speech separation criterion (i.e., Si-SNR) directly. We emphasize on how tied kernel functions for calculating spatial features, encoder, and decoder in multi-channel framework can be effective. We simulate spatialized reverberate data for both WSJ0 and LibriSpeech corpora here, and while these two sets of data are different in the matter of size and duration, the effect of capturing shorter and longer dependencies of previous/+future samples are studied in detail. We report SDR, Si-SNR and PESQ to evaluate the performance of developed solutions.

Motivation & Objective

• To develop a unified deep learning framework that integrates both spectrogram-based and waveform-based speech separation into a single architecture.
• To enable end-to-end training for both single-channel and multi-channel speech separation while optimizing for Si-SNR.
• To investigate the impact of spatial features and tied kernel functions in multi-channel setups on separation performance.
• To evaluate the robustness and stability of the framework across diverse datasets, including simulated reverberant WSJ0-2mix and LibriSpeech-2mix.
• To compare the performance, memory efficiency, and latency trade-offs between spectrogram-based and waveform-based speech separation pipelines.

Proposed method

• The framework uses a shared encoder-decoder structure with different kernel functions to process either spectrogram or raw waveform inputs.
• It employs tied kernel functions for spatial feature extraction in multi-channel setups to improve generalization and parameter efficiency.
• End-to-end training is performed using the scale-invariant signal-to-noise ratio (Si-SNR) as the optimization criterion.
• The model is trained on simulated spatialized, reverberant speech data for both WSJ0-2mix and LibriSpeech-2mix corpora.
• Latency is controlled via configurable receptive field and kernel size, enabling low-latency inference with minimal performance degradation.
• Phase information from the mixed signal is preserved in the reconstruction stage, even in spectrogram-based pipelines where phase is not updated during training.

Experimental results

Research questions

• RQ1 Can a unified deep learning framework effectively combine spectrogram-based and waveform-based speech separation under a single architecture?
• RQ2 How does multi-channel speech separation with spatial feature learning compare to single-channel setups in terms of performance and robustness?
• RQ3 What is the trade-off between memory efficiency, latency, and performance between spectrogram-based and waveform-based separation?
• RQ4 How do different network structures and hyper-parameters affect the stability and generalization of the separation model?
• RQ5 Does using tied kernel functions for spatial feature extraction improve performance in multi-channel speech separation?

Key findings

• Spectrogram-based separation outperformed waveform-based separation in both single- and multi-channel setups, even when phase was not updated during training.
• The proposed multi-channel framework consistently outperformed single-channel baselines across all evaluation metrics, including SDR, Si-SNR, and PESQ.
• Spectrogram-based models showed greater stability across different network architectures and hyper-parameter settings compared to waveform-based models.
• The unified framework achieved competitive performance with reduced memory usage, especially when using large STFT windows (e.g., L=512).
• PESQ scores aligned with SDR and Si-SNR trends, confirming perceptual quality improvements across all evaluated pipelines.
• The framework enabled low-latency inference with negligible performance degradation through configurable kernel and receptive field settings.