[Paper Review] The DKU-DukeECE-Lenovo System for the Diarization Task of the 2021 VoxCeleb Speaker Recognition Challenge

Weiqing Wang, Danwei Cai+5 more|arXiv (Cornell University)|Sep 5, 2021

Speech Recognition and SynthesisComputer Science23 references19 citations

TL;DR

This paper presents a hybrid speaker diarization system for the 2021 VoxCeleb Speaker Recognition Challenge, integrating voice activity detection (VAD), speaker embedding, clustering, and novel overlap detection methods. The key innovation is a 2-speaker target-speaker voice activity detection (TS-VAD) model that identifies overlapping speech between speaker pairs, significantly reducing missed speaker errors and achieving a 5.07% diarization error rate (DER) on the test set, ranking first in the challenge.

ABSTRACT

This report describes the submission of the DKU-DukeECE-Lenovo team to the VoxCeleb Speaker Recognition Challenge (VoxSRC) 2021 track 4. Our system including a voice activity detection (VAD) model, a speaker embedding model, two clustering-based speaker diarization systems with different similarity measurements, two different overlapped speech detection (OSD) models, and a target-speaker voice activity detection (TS-VAD) model. Our final submission, consisting of 5 independent systems, achieves a DER of 5.07% on the challenge test set.

Motivation & Objective

• To reduce diarization error rate (DER) in real-world, overlapping speech scenarios by improving overlap detection.
• To address missed speaker errors caused by conventional diarization systems failing to detect overlapping speech.
• To develop a flexible, speaker-agnostic overlap detection method that does not assume a fixed number of speakers.
• To enhance system robustness through multi-model fusion and data augmentation.
• To achieve state-of-the-art performance in the VoxSRC 2021 challenge, particularly in handling overlapping speech.

Proposed method

• A ResNet34-BiLSTM architecture is used for VAD, overlap detection, and TS-VAD, with statistical pooling and sigmoid output for frame-level speech probability.
• A 2-speaker TS-VAD model is proposed, where pairs of speaker embeddings are fed into a TS-VAD network to detect overlapping speech regions between two speakers, enabling detection without prior speaker count constraints.
• Speaker embeddings are extracted using a ResNet34 front-end with global statistic pooling and ArcFace loss, trained on VoxCeleb 1 & 2 for robustness.
• Agglomerative hierarchical clustering (AHC) and LSTM-based similarity with spectral clustering are used for speaker clustering, with hyperparameters tuned on DEV402.
• A multi-stage training strategy is employed for TS-VAD: pre-training on simulated Librispeech, transfer to VoxConverse, and fine-tuning on DEV402 with data augmentation using MUSAN and RIRs.
• System fusion via DOVER-Lap with rank-based weighting combines five independent systems, improving robustness and final DER.

Experimental results

Research questions

• RQ1 Can a 2-speaker TS-VAD model effectively detect overlapping speech regions between any pair of speakers without assuming a fixed number of speakers?
• RQ2 How does integrating multiple overlap detection methods (VAD, TS-VAD, 2-speaker TS-VAD) improve overall diarization performance?
• RQ3 To what extent can data augmentation with MUSAN and RIRs improve model generalization in real-world diarization settings?
• RQ4 Does DOVER-Lap fusion of multiple diverse systems lead to better generalization and lower DER than individual systems?
• RQ5 Can a multi-stage training and transfer learning strategy effectively adapt a TS-VAD model from simulated to real-world data?

Key findings

• The proposed 2-speaker TS-VAD model successfully detects overlapping speech between any pair of speakers, enabling detection without prior knowledge of speaker count.
• The final system, fusing five independent models via DOVER-Lap with rank-based weighting, achieved a 5.07% diarization error rate (DER) on the VoxSRC 2021 test set, ranking first in the challenge.
• The 2-speaker TS-VAD method reduced missed speaker errors by accurately identifying overlapping regions, contributing significantly to the overall performance gain.
• System fusion with DOVER-Lap improved robustness and generalization, outperforming individual systems and earlier fusion attempts that showed signs of overfitting on the validation set.
• The multi-stage training strategy—pre-training on Librispeech, transfer to VoxConverse, and fine-tuning on DEV402—enabled effective adaptation of the TS-VAD model to real-world conditions.
• Data augmentation using MUSAN and RIRs improved model robustness across diverse acoustic conditions, particularly benefiting VAD and TS-VAD models.