[Paper Review] Target Speaker Extraction for Overlapped Multi-Talker Speaker Verification

Wei Rao, Chenglin Xu+2 more|arXiv (Cornell University)|Feb 7, 2019

Speech Recognition and Synthesis15 references4 citations

TL;DR

This paper proposes a target speaker extraction framework for overlapped multi-talker speaker verification, using a target speaker extraction module to isolate the target speaker's speech from mixed utterances before feeding it into an i-vector/PLDA speaker verification system. The method achieves a 65.7% relative EER reduction, with SBF-MTSAL-Concat outperforming SBF-MTSAL.

ABSTRACT

The performance of speaker verification degrades significantly when the test speech is corrupted by interference speakers. Speaker diarization does well to separate speakers if the speakers are temporally overlapped. However, if multi-talkers speak at the same time, we need the technique to separate the speech in the spectral domain. This paper proposes an overlapped multi-talker speaker verification framework by using target speaker extraction methods. Specifically, given the target speaker information, the target speaker's speech is firstly extracted from the overlapped multi-talker speech by a target speaker extraction module. Then, the extracted speech is passed to the speaker verification system. Experimental results show that the proposed approach significantly improves the performance of overlapped multi-talker speaker verification and achieves 65.7% relative EER reduction.

Motivation & Objective

• To address the degradation in speaker verification performance when multiple speakers speak simultaneously.
• To overcome the limitation of existing speech separation methods that require prior knowledge of the number of speakers.
• To improve speaker verification robustness in fully overlapped multi-talker scenarios using target speaker extraction.
• To compare the effectiveness of two target speaker extraction networks—SBF-MTSAL and SBF-MTSAL-Concat—in overlapped multi-talker settings.
• To mitigate the mismatch between extracted and clean speech in speaker verification by combining clean and extracted training data.

Proposed method

• The framework uses a target speaker extraction module that leverages the enrollment utterance as auxiliary information to isolate the target speaker’s speech from a mixed utterance.
• The target speaker extraction network employs magnitude and temporal spectrum approximation loss to estimate a phase-sensitive mask for improved speech reconstruction.
• SBF-MTSAL uses an auxiliary network to generate adaptation weights that refine mask estimation in a CADNN-based architecture.
• SBF-MTSAL-Concat enhances SBF-MTSAL by concatenating the auxiliary and mixture features before mask estimation, improving representation learning.
• The extracted speech is then processed by a standard i-vector/PLDA speaker verification system for verification.
• A combined training set (clean+ext) is created by pooling extracted speech from development data to reduce mismatch between extracted and clean speech.

Experimental results

Research questions

• RQ1 Can target speaker extraction significantly improve speaker verification performance in fully overlapped multi-talker scenarios?
• RQ2 How does the performance of SBF-MTSAL compare to SBF-MTSAL-Concat in overlapped multi-talker speaker verification?
• RQ3 To what extent does combining clean and extracted speech in training reduce the mismatch between extracted and clean speech in speaker verification?
• RQ4 Does the proposed framework maintain robustness when the test speech is fully overlapped with multiple speakers?
• RQ5 What is the upper bound performance of target speaker extraction in this setting, and how close does the proposed method come to it?

Key findings

• The proposed target speaker extraction framework achieves a 65.7% relative reduction in equal error rate (EER) compared to the baseline system on fully overlapped test data.
• SBF-MTSAL-Concat outperforms SBF-MTSAL, achieving better EER and DCF performance in overlapped multi-talker speaker verification.
• Using a combined clean+ext training set improves performance on the mixture test set when using SBF-MTSAL-Concat, indicating effective mismatch mitigation.
• The clean+ext training set improves EER on clean test sets but degrades DCF performance, suggesting trade-offs in generalization.
• The baseline system shows significant performance degradation on fully overlapped test data, highlighting the challenge of multi-talker interference.
• The upper bound performance (ideal extraction) shows that the proposed method is close to optimal, indicating strong potential for real-world deployment.