[Paper Review] Speaker-Conditional Chain Model for Speech Separation and Extraction

Jing Shi, Jiaming Xu+3 more|arXiv (Cornell University)|Jun 25, 2020

Speech and Audio Processing35 references4 citations

TL;DR

This paper proposes a Speaker-Conditional Chain Model (SCCM) that jointly infers speaker identities and extracts speech sources from fully overlapped, long-form audio recordings with variable numbers of speakers. By using a sequence-to-sequence speaker inference module followed by speaker-conditional speech extraction, SCCM achieves stable performance across multi-round conversations and outperforms baseline models in both SI-SNRi and WER metrics on long and overlapping recordings.

ABSTRACT

Speech separation has been extensively explored to tackle the cocktail party problem. However, these studies are still far from having enough generalization capabilities for real scenarios. In this work, we raise a common strategy named Speaker-Conditional Chain Model to process complex speech recordings. In the proposed method, our model first infers the identities of variable numbers of speakers from the observation based on a sequence-to-sequence model. Then, it takes the information from the inferred speakers as conditions to extract their speech sources. With the predicted speaker information from whole observation, our model is helpful to solve the problem of conventional speech separation and speaker extraction for multi-round long recordings. The experiments from standard fully-overlapped speech separation benchmarks show comparable results with prior studies, while our proposed model gets better adaptability for multi-round long recordings.

Motivation & Objective

• To address the limitations of existing speech separation models that assume fixed speaker counts and struggle with long, multi-round recordings.
• To enable joint inference of speaker identities and extraction of individual speech sources in fully overlapped, long-form audio without prior knowledge of speaker count.
• To improve generalization and stability in real-world scenarios with sparse overlaps (e.g., ~15% overlap) and dynamic speaker roles across segments.
• To provide a unified framework that supports downstream tasks like speech recognition by delivering both clean speech sources and speaker identity labels.
• To overcome the limitations of conventional models that fail to maintain consistent speaker ordering across segments in long recordings.

Proposed method

• The model uses a sequence-to-sequence architecture to infer speaker identities from the entire spectrogram input, producing a sequence of speaker embeddings.
• The inferred speaker identities are then used as conditional inputs to a speech extraction module that generates individual speech sources.
• The framework is trained end-to-end using a chain rule of conditional probabilities: P(sources | mixture) = P(speakers | mixture) × P(sources | mixture, speakers).
• The speaker inference module processes the full recording, while the extraction module operates on 4-second random segments to avoid memory overflow.
• The model is trained on synthetic multi-round recordings created by concatenating utterances from different speakers with random temporal shifts to simulate natural conversation dynamics.
• The architecture supports open-vocabulary speaker tasks by learning speaker-related hidden representations rather than relying on fixed speaker embeddings.

Figure 1: The framework of the proposed Speaker-Conditional Chain Model (SCCM). (a) shows the whole strategy of our proposed SCCM; (b) is the module of speaker inference , which predicts the speaker identities and corresponding embeddings. (c) refers to the time-domain speech extraction module. This

Experimental results

Research questions

• RQ1 Can a unified model jointly infer speaker identities and extract speech sources from long, fully overlapped recordings with variable speaker counts?
• RQ2 How does the proposed speaker-conditional chain model perform compared to baseline separation models on long recordings with multi-round conversations?
• RQ3 Does the model maintain consistent speaker ordering and performance across segments when speaker roles change over time?
• RQ4 To what extent does the model generalize to real-world scenarios with low overlap ratios (~15%) and natural speaker turn-taking?
• RQ5 Can the speaker identity information produced by the model be effectively used to improve downstream tasks like automatic speech recognition?

Key findings

• SCCM achieved a test SI-SNRi of 13.7 on multi-round recordings, outperforming the baseline TasNet model (11.5), demonstrating improved stability and performance in long-form settings.
• On the same multi-round benchmark, SCCM reduced word error rate (WER) by 10.2% on average compared to the baseline model at 40% overlap, showing robustness to increasing overlap.
• The model maintained a 12.6% WER on non-overlapped segments (0S) and 26.8% on high-overlap segments (40%), indicating consistent performance across varying overlap conditions.
• The speaker inference module successfully captured speaker activity patterns across time, with attention visualization showing alignment with speaker turns in the recording.
• The model demonstrated better adaptability to long recordings than conventional models, which struggle with speaker ordering inconsistencies across segments.
• The inclusion of speaker identity information as a conditional input improved downstream speech recognition performance, especially in high-overlap scenarios.

Figure 2: Visualization of one sample of the learned attention status in speaker inference module for overlapped speech in WSJ0-2mix.