[Paper Review] Single-Channel Multi-talker Speech Recognition with Permutation Invariant Training

Yanmin Qian, Xuankai Chang+1 more|arXiv (Cornell University)|Jul 19, 2017

Speech Recognition and Synthesis3 citations

TL;DR

This paper proposes a permutation invariant training (PIT)-based approach for single-channel multi-talker speech recognition, directly optimizing for speaker separation and recognition in a joint end-to-end framework. By applying PIT to both front-end feature separation (using MSE) and back-end ASR (using cross-entropy), the method effectively resolves label permutation issues, achieving 45.0% and 25.0% relative WER reductions on two- and three-talker mixed speech, respectively, with strong generalization across speaker counts.

ABSTRACT

Although great progresses have been made in automatic speech recognition (ASR), significant performance degradation is still observed when recognizing multi-talker mixed speech. In this paper, we propose and evaluate several architectures to address this problem under the assumption that only a single channel of mixed signal is available. Our technique extends permutation invariant training (PIT) by introducing the front-end feature separation module with the minimum mean square error (MSE) criterion and the back-end recognition module with the minimum cross entropy (CE) criterion. More specifically, during training we compute the average MSE or CE over the whole utterance for each possible utterance-level output-target assignment, pick the one with the minimum MSE or CE, and optimize for that assignment. This strategy elegantly solves the label permutation problem observed in the deep learning based multi-talker mixed speech separation and recognition systems. The proposed architectures are evaluated and compared on an artificially mixed AMI dataset with both two- and three-talker mixed speech. The experimental results indicate that our proposed architectures can cut the word error rate (WER) by 45.0% and 25.0% relatively against the state-of-the-art single-talker speech recognition system across all speakers when their energies are comparable, for two- and three-talker mixed speech, respectively. To our knowledge, this is the first work on the multi-talker mixed speech recognition on the challenging speaker-independent spontaneous large vocabulary continuous speech task.

Motivation & Objective

• To address the challenge of recognizing multi-talker mixed speech from a single-channel input, where speaker identities are ambiguous.
• To overcome the label permutation problem in deep learning-based multi-talker ASR systems, which hinders reliable training and inference.
• To develop a unified end-to-end architecture that jointly optimizes feature separation and speech recognition without explicit separation stages.
• To evaluate the model on speaker-independent, spontaneous large-vocabulary continuous speech tasks under realistic conditions.
• To demonstrate generalization capability across varying numbers of speakers, including using a three-speaker model to recognize two-speaker mixtures.

Proposed method

• Applies permutation invariant training (PIT) to both front-end feature separation and back-end ASR, using minimum mean square error (MSE) and minimum cross-entropy (CE) criteria, respectively.
• For each utterance, computes the loss over all possible output-target speaker assignments and selects the one with the minimum error for optimization.
• Introduces a direct PIT-CE-ASR model that bypasses explicit speech separation by directly predicting senone posteriors for each speaker.
• Uses a joint-optimization architecture that combines front-end separation and back-end recognition under a unified training objective.
• Employs a single-channel mixed speech input and leverages deep neural networks to model speaker-specific representations without prior speaker information.
• Evaluates the model on artificially mixed AMI datasets with both two- and three-talker scenarios, varying speaker energy levels and gender combinations.

Experimental results

Research questions

• RQ1 Can permutation invariant training be effectively extended from speech separation to end-to-end multi-talker speech recognition?
• RQ2 Does joint optimization of feature separation and ASR via PIT improve recognition performance compared to separate training stages?
• RQ3 How does the proposed model generalize when applied to mixed speech with a different number of speakers than it was trained on?
• RQ4 What is the performance gain of the proposed method over state-of-the-art single-talker ASR systems in challenging multi-talker scenarios?
• RQ5 Does the model maintain robustness across different speaker energy levels and gender combinations?

Key findings

• The proposed direct PIT-CE-ASR model achieves a 45.0% relative word error rate (WER) reduction compared to a state-of-the-art single-talker ASR system on two-talker mixed speech with comparable speaker energies.
• For three-talker mixed speech, the model achieves a 25.0% relative WER reduction across all speakers under the same energy condition.
• The three-talker PIT-CE-ASR model generalizes well to two-talker mixed speech, achieving WERs nearly identical to a dedicated two-speaker model, indicating robustness to speaker count variation.
• Performance is better on mixed speech with different genders (e.g., 64.80% WER on different-gender three-talker mix) than on same-gender mixtures (e.g., 69.78% WER), suggesting gender diversity aids recognition.
• The model maintains strong performance even at low SNR conditions, with WERs of 72.88% and 81.63% at 15dB and 20dB SNR, respectively, for three-talker mixtures.
• To the authors' knowledge, this is the first successful end-to-end recognition system for speaker-independent, spontaneous, large-vocabulary continuous speech in multi-talker settings with single-channel input.