[Paper Review] The DKU-DukeECE Diarization System for the VoxCeleb Speaker Recognition Challenge 2022

Weiqing Wang, Xiaoyi Qin+4 more|arXiv (Cornell University)|Oct 4, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper presents the DKU-DukeECE system for the VoxCeleb Speaker Recognition Challenge 2022, Track 4, which achieves a 4.75% diarization error rate (DER) by fusing a multi-model voice activity detection (VAD), target-speaker VAD (TS-VAD) for overlap detection, and an improved speaker embedding model within an agglomerative hierarchical clustering (AHC) framework. The system ranks first by leveraging DOVER-Lap fusion of four distinct configurations, significantly reducing DER through enhanced VAD and overlap handling.

ABSTRACT

This paper discribes the DKU-DukeECE submission to the 4th track of the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22). Our system contains a fused voice activity detection model, a clustering-based diarization model, and a target-speaker voice activity detection-based overlap detection model. Overall, the submitted system is similar to our previous year's system in VoxSRC-21. The difference is that we use a much better speaker embedding and a fused voice activity detection, which significantly improves the performance. Finally, we fuse 4 different systems using DOVER-lap and achieve 4.75 of the diarization error rate, which ranks the 1st place in track 4.

Motivation & Objective

• To improve speaker diarization performance in multi-speaker, overlapped speech scenarios using robust voice activity detection (VAD) and overlap detection.
• To reduce diarization error rate (DER) by enhancing VAD models and integrating target-speaker VAD (TS-VAD) for better overlap region detection.
• To mitigate domain mismatch between training (VoxCeleb) and test (VoxConverse) sets through pseudo-labeling and fine-tuning of speaker embeddings.
• To improve system robustness and generalization by fusing multiple diarization systems using DOVER-Lap.
• To achieve state-of-the-art performance in the VoxSRC 2022 Speaker Recognition Challenge, Track 4.

Proposed method

• Fuses four distinct VAD models—ResNet34-based, Conformer-based, pyannote.audio 2.0, and ASR-based—using majority voting to reduce false alarms and missed detections.
• Employs a SimAM-ResNet34-based speaker embedding model with ArcFace loss, fine-tuned using pseudo-labeled VoxConverse dev set to reduce domain gap.
• Introduces a target-speaker VAD (TS-VAD) model that uses speaker embeddings to detect speech segments of specific targets, trained on simulated Librispeech data and fine-tuned on VoxConverse.
• Applies DOVER-Lap fusion to combine four system variants: AHC with overlap detection, AHC with partially/fully assigned TS-VAD, and LSTM-based spectral clustering with TS-VAD.
• Uses online data augmentation with MUSAN and RIRs to improve robustness across diverse acoustic conditions.
• Applies a hybrid inference strategy: AHC-based results are partially corrected using TS-VAD outputs to reduce speaker confusion and overestimation.

Experimental results

Research questions

• RQ1 How can multiple VAD models be effectively fused to reduce diarization error rate (DER) in multi-speaker scenarios?
• RQ2 To what extent does target-speaker VAD (TS-VAD) improve overlap detection and overall diarization performance compared to standard VAD or overlap-only detection?
• RQ3 Can speaker embedding models pre-trained on VoxCeleb be effectively adapted to the VoxConverse test set to reduce domain mismatch and improve DER?
• RQ4 How does DOVER-Lap fusion of diverse diarization systems enhance robustness and lower DER in the presence of speaker confusion and overestimation?
• RQ5 What is the impact of data augmentation using MUSAN and RIRs on the generalization of VAD and diarization models?

Key findings

• The fused VAD system reduced DER by 0.3% compared to individual VAD models, achieving 3.97% error rate on the VoxConverse test set.
• The AHC-based diarization system with TS-VAD achieved 4.85% DER on both the VoxConverse and challenge test sets, ranking first in the leaderboard.
• Fusing four distinct systems via DOVER-Lap reduced the DER to 4.75%, securing the first-place ranking in Track 4 of VoxSRC 2022.
• The TS-VAD model reduced DER by approximately 0.6% compared to baseline AHC with standard overlap detection, demonstrating its effectiveness in handling overlapping speech.
• The system significantly outperformed the AHC-only baseline, which overestimated speaker count (e.g., predicting 30+ speakers for a 10-speaker segment), by using partial TS-VAD assignment to correct errors.
• Despite fine-tuning on VoxConverse, the speaker embedding model performed worse on the challenge test set than the pre-trained VoxCeleb2 model, so the latter was used for final inference.