[Paper Review] Single-Channel Speech Separation with Auxiliary Speaker Embeddings

Shuo Liu, Gil Keren+1 more|arXiv (Cornell University)|Jun 24, 2019

Speech and Audio Processing23 references4 citations

TL;DR

This paper proposes a single-channel speech separation model that improves performance by using auxiliary speaker embeddings derived from clean context recordings of each speaker. By conditioning the separation network on these embeddings, the model better attributes time-frequency bins to the correct speaker, achieving 4.79 dB SDR, 8.44 dB SAR, and 7.11 dB SIR on the challenging VoxCeleb dataset—surpassing state-of-the-art baselines like deep clustering and DaNet.

ABSTRACT

We present a novel source separation model to decompose asingle-channel speech signal into two speech segments belonging to two different speakers. The proposed model is a neural network based on residual blocks, and uses learnt speaker embeddings created from additional clean context recordings of the two speakers as input to assist in attributing the different time-frequency bins to the two speakers. In experiments, we show that the proposed model yields good performance in the source separation task, and outperforms the state-of-the-art baselines. Specifically, separating speech from the challenging VoxCeleb dataset, the proposed model yields 4.79dB signal-to-distortion ratio, 8.44dB signal-to-artifacts ratio and 7.11dB signal-to-interference ratio.

Motivation & Objective

• To address the challenge of single-channel speech separation when speakers have similar acoustic features.
• To improve source separation performance by conditioning the model on speaker embeddings derived from clean context recordings.
• To reduce ambiguity in embedding space clustering by providing explicit speaker-specific reference embeddings.
• To enhance separation quality in real-world settings where clean speech segments from each speaker are available.
• To outperform existing deep clustering and deep attractor network baselines on a large-scale, natural speech dataset.

Proposed method

• The model uses a stack of residual blocks for both the separation network and speaker embedding subnetworks.
• Separate speaker embedding subnetworks process clean context recordings of each speaker to generate speaker-specific embeddings.
• The main separation network processes the mixed speech segment and is conditioned on both speaker embeddings to guide bin attribution.
• The network outputs an estimated interference frame, and the difference with the central frame yields the estimated target frame.
• The model is trained to minimize reconstruction loss on the target signal, with embeddings used as conditional inputs to improve separation accuracy.
• At inference, the model uses the same speaker embeddings to condition separation, enabling robust performance on unseen speakers.

Experimental results

Research questions

• RQ1 Can auxiliary speaker embeddings from clean context recordings improve single-channel speech separation performance?
• RQ2 How does conditioning the separation network on speaker embeddings reduce ambiguity in time-frequency bin attribution?
• RQ3 Does the proposed method outperform deep clustering and deep attractor networks on natural, diverse speech data?
• RQ4 To what extent do speaker embeddings enhance separation quality when speakers have similar acoustic features?
• RQ5 Can the model generalize to unseen speakers using only short clean context recordings?

Key findings

• The proposed model achieves 4.79 dB signal-to-distortion ratio (SDR), significantly outperforming the deep clustering baseline (0.84 dB) and DaNet (1.81 dB).
• The model attains 8.44 dB signal-to-artifacts ratio (SAR), indicating superior speech quality and reduced algorithmic distortions compared to baselines.
• With 7.11 dB signal-to-interference ratio (SIR), the model effectively suppresses interference, though DaNet achieved a higher SIR (10.41 dB), suggesting trade-offs in target preservation.
• The performance gain is attributed to the use of auxiliary speaker embeddings, which provide explicit reference points for embedding space clustering.
• The model demonstrates strong generalization to unseen speakers on the large-scale VoxCeleb dataset, indicating robustness in real-world settings.
• Visual results confirm that the model successfully separates target and interference speakers, preserving speech intelligibility and reducing artifacts.