[Paper Review] Multi-talker Speech Separation and Tracing with Permutation Invariant Training of Deep Recurrent Neural Networks.

Morten Kolbæk, Dong Yu+2 more|arXiv (Cornell University)|Mar 18, 2017

Speech and Audio Processing27 references13 citations

TL;DR

This paper introduces utterance-level Permutation Invariant Training (uPIT), an end-to-end deep learning method using RNNs that minimizes utterance-level separation error to eliminate the need for permutation alignment during inference. uPIT enables speaker-independent multi-talker speech separation without prior knowledge of speaker count, identity, or gender, outperforming NMF, CASA, and matching DPCL and DANet performance while generalizing across unseen speakers and languages.

ABSTRACT

In this paper we propose the utterance-level Permutation Invariant Training (uPIT) technique. uPIT is a practically applicable, end-to-end, deep learning based solution for speaker independent multi-talker speech separation. Specifically, uPIT extends the recently proposed Permutation Invariant Training (PIT) technique with an utterance-level cost function, hence eliminating the need for solving an additional permutation problem during inference, which is otherwise required by frame-level PIT. We achieve this using Recurrent Neural Networks (RNNs) that, during training, minimize the utterance-level separation error, hence forcing separated frames belonging to the same speaker to be aligned to the same output stream. In practice, this allows RNNs, trained with uPIT, to separate multi-talker mixed speech without any prior knowledge of signal duration, number of speakers, speaker identity or gender. We evaluated uPIT on the WSJ0 and Danish two- and three-talker mixed-speech separation tasks and found that uPIT outperforms techniques based on Non-negative Matrix Factorization (NMF) and Computational Auditory Scene Analysis (CASA), and compares favorably with Deep Clustering (DPCL) and the Deep Attractor Network (DANet). Furthermore, we found that models trained with uPIT generalize well to unseen speakers and languages. Finally, we found that a single model, trained with uPIT, can handle both two-speaker, and three-speaker speech mixtures.

Motivation & Objective

• To address the challenge of speaker-independent multi-talker speech separation without requiring prior knowledge of speaker count, identity, or gender.
• To eliminate the need for solving the permutation problem during inference, which is required in frame-level Permutation Invariant Training (PIT).
• To develop a unified model capable of handling both two- and three-speaker mixtures without retraining.
• To improve generalization to unseen speakers and languages compared to existing methods.

Proposed method

• uPIT extends frame-level PIT by introducing an utterance-level cost function that minimizes the total separation error across the entire utterance, not per frame.
• Recurrent Neural Networks (RNNs) are trained to align frames from the same speaker to the same output stream by minimizing the utterance-level error.
• The method avoids the permutation problem at inference by learning speaker-specific stream alignment during training through the utterance-level objective.
• The model is trained end-to-end on mixed speech data without speaker identity annotations, relying solely on the utterance-level loss to enforce consistent speaker separation.
• The approach is applied to WSJ0 and Danish datasets with two- and three-speaker mixtures, using RNNs to model sequential dependencies in speech.
• The training objective ensures that the same speaker's frames are consistently assigned to the same output stream, regardless of permutation ambiguity.

Experimental results

Research questions

• RQ1 Can an end-to-end deep learning model achieve speaker-independent multi-talker speech separation without solving the permutation problem at inference?
• RQ2 Does training with an utterance-level loss improve generalization to unseen speakers and languages compared to frame-level PIT?
• RQ3 Can a single uPIT model effectively separate both two-speaker and three-speaker mixtures without retraining?
• RQ4 How does uPIT compare in performance to NMF, CASA, DPCL, and DANet on standard multi-talker separation benchmarks?

Key findings

• uPIT outperforms Non-negative Matrix Factorization (NMF) and Computational Auditory Scene Analysis (CASA) on both WSJ0 and Danish datasets.
• uPIT achieves performance comparable to Deep Clustering (DPCL) and the Deep Attractor Network (DANet) on the same benchmarks.
• Models trained with uPIT generalize well to unseen speakers and languages, demonstrating robustness beyond the training distribution.
• A single uPIT model trained on both two- and three-speaker mixtures successfully handles both scenarios without retraining or architectural changes.
• The utterance-level training objective effectively resolves the permutation ambiguity during inference, eliminating the need for post-processing permutation matching.
• The method achieves state-of-the-art performance among end-to-end approaches without requiring speaker identity or gender information during training.