[Paper Review] The DKU-DukeECE Systems for VoxCeleb Speaker Recognition Challenge 2020

Weiqing Wang, Danwei Cai+2 more|arXiv (Cornell University)|Oct 24, 2020

Speech Recognition and SynthesisComputer Science27 references22 citations

TL;DR

This paper presents the DKU-DukeECE systems for the VoxCeleb Speaker Recognition Challenge 2020, introducing advanced speaker verification and diarization pipelines. For track 1, it combines ResNet, ResNet-BAM, and ECAPA-TDNN with score normalization and fusion, achieving 2.48% EER on VoxSRC20-dev; for track 4, it employs iterative self-supervised learning, VAD, uniform segmentation, self-attention similarity modeling, and spectral clustering, reducing DER to 6.46% and JER to 28.58% on dev-66 after finetuning.

ABSTRACT

In this paper, we present the system submission for the VoxCeleb Speaker Recognition Challenge 2020 (VoxSRC-20) by the DKU-DukeECE team. For track 1, we explore various kinds of state-of-the-art front-end extractors with different pooling layers and objective loss functions. For track 3, we employ an iterative framework for self-supervised speaker representation learning based on a deep neural network (DNN). For track 4, we investigate the whole system pipeline for speaker diarization, including voice activity detection (VAD), uniform segmentation, speaker embedding extraction, and clustering.

Motivation & Objective

• To develop high-performance speaker verification and diarization systems for the VoxCeleb Speaker Recognition Challenge 2020.
• To improve robustness and generalization through data augmentation, including noise, reverberation, and speed perturbation.
• To explore iterative self-supervised learning for speaker representation learning in track 3.
• To optimize the full diarization pipeline, including VAD, segmentation, embedding extraction, similarity modeling, and clustering.
• To achieve state-of-the-art performance on VoxCeleb and VoxConverse dev sets using end-to-end system integration and finetuning.

Proposed method

• For track 1, the system uses ResNet, ResNet-BAM, and ECAPA-TDNN with 80-dimensional log Mel-filterbank features, global statistic pooling, and ArcFace loss for speaker embedding learning.
• The system applies adaptive symmetric score normalization (AS-Norm) and score-level fusion with calibrated weights (1, 1.2, 1) for ResNet, ResNet-BAM, and ECAPA-TDNN.
• For track 3, an iterative self-supervised framework trains a DNN using pseudo-labels from clustering, improving speaker embeddings over multiple rounds.
• For track 4, the pipeline includes uniform segmentation (1.5s with 0.75s shift), ResNet-based speaker embedding extraction with softmax loss, and self-attention-based similarity modeling to compute affinity matrices.
• The similarity module uses a multi-head attention network with 256-dimensional input, 128-unit heads, and a 1024-dimensional feed-forward layer, outputting a sigmoid-scaled affinity score.
• Spectral clustering with symmetrized and normalized affinity matrices is used for diarization, with speaker count estimated via a 0.99 threshold.

Experimental results

Research questions

• RQ1 How do different front-end extractors (ResNet, ResNet-BAM, ECAPA-TDNN) compare in speaker verification under varying test conditions?
• RQ2 Can iterative self-supervised learning with pseudo-labeling improve speaker representation learning in low-resource settings?
• RQ3 How effective is self-attention-based similarity modeling in capturing speaker relationships across short segments for diarization?
• RQ4 What is the impact of finetuning VAD, similarity, and overlap detection models on diarization performance?
• RQ5 Can a unified pipeline combining VAD, segmentation, embedding, similarity, and clustering achieve state-of-the-art results on VoxConverse?

Key findings

• The fused system of ResNet, ResNet-BAM, and ECAPA-TDNN achieved 2.48% EER and 0.1252 mDCF 0.05 on the VoxSRC20-dev set, outperforming individual models.
• The iterative self-supervised training in track 3 reduced minDCF from 0.857 to 0.479 and EER from 20.11% to 9.60% on VoxCeleb1-H after two rounds.
• After finetuning on 150 dev recordings, the diarization system achieved 6.46% DER and 28.58% JER on the last 66 recordings of the dev set.
• The DER on the full dev set (dev-all) was 9.74% without finetuning, improving to 6.46% with finetuning on a held-out subset.
• The self-attention similarity model significantly improved affinity estimation, enabling accurate speaker clustering despite variable segment lengths.
• Overlap detection with boundary extension improved robustness, achieving 15.7% JER on dev-66 with finetuning.