[Paper Review] Self-Supervised Learning with Cluster-Aware-DINO for High-Performance Robust Speaker Verification

Bing Han, Zhengyang Chen+1 more|arXiv (Cornell University)|Apr 12, 2023

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes Cluster-Aware DINO (CA-DINO), a self-supervised speaker verification framework that leverages the DINO distillation objective to eliminate reliance on negative pairs and enhances robustness via cluster-aware training and dynamic loss-gate with label correction (DLG-LC). It achieves state-of-the-art performance on VoxCeleb, outperforming fully supervised systems in some settings without using any human-annotated labels.

ABSTRACT

Automatic speaker verification task has made great achievements using deep learning approaches with the large-scale manually annotated dataset. However, it's very difficult and expensive to collect a large amount of well-labeled data for system building. In this paper, we propose a novel and advanced self-supervised learning framework which can construct a high performance speaker verification system without using any labeled data. To avoid the impact of false negative pairs, we adopt the self-distillation with no labels (DINO) framework as the initial model, which can be trained without exploiting negative pairs. Then, we introduce a cluster-aware training strategy for DINO to improve the diversity of data. In the iteration learning stage, due to a mass of unreliable labels from clustering, the quality of pseudo labels is important for the system training. This motivates us to propose dynamic loss-gate and label correction (DLG-LC) methods to alleviate the performance degradation caused by unreliable labels. More specifically, we model the loss distribution with GMM and obtain the loss-gate threshold dynamically to distinguish the reliable and unreliable labels. Besides, we adopt the model predictions to correct the unreliable label, for better utilizing the unreliable data rather than dropping them directly. Moreover, we extend the DLG-LC to multi-modality to further improve the performance. The experiments are performed on the commonly used Voxceleb dataset. Compared to the best-known self-supervised speaker verification system, our proposed method obtain 22.17%, 27.94% and 25.56% relative EER improvement on Vox-O, Vox-E and Vox-H test sets, even with fewer iterations, smaller models, and simpler clustering methods. More importantly, the newly proposed system even achieves comparable results with the fully supervised system, but without using any human labeled data.

Motivation & Objective

• Address the high cost and scarcity of large-scale, well-labeled speaker verification datasets by enabling fully self-supervised training.
• Overcome limitations of contrastive learning in self-supervised SV, particularly false negative pairs from cross-utterance speaker mismatches.
• Improve model robustness and performance in iterative self-supervised training by enhancing pseudo-label quality despite unreliable clustering.
• Reduce the performance gap between self-supervised and fully supervised speaker verification systems through advanced loss and label handling.

Proposed method

• Adopt the DINO self-distillation framework as the initial pre-training stage, eliminating the need for negative pairs and enabling training without explicit contrastive objectives.
• Introduce cluster-aware training by collecting positive samples from the same speaker cluster across multiple utterances, increasing data diversity and representation quality.
• Implement a dynamic loss-gate (DLG) mechanism that models the loss distribution using a Gaussian Mixture Model (GMM) to identify and filter unreliable pseudo-labeled samples.
• Apply label correction (LC) by replacing unreliable pseudo-labels with model-predicted posterior distributions to preserve information and avoid data loss.
• Extend the DLG-LC framework to multi-modal learning by fusing audio and visual features during iterative training to further improve robustness and accuracy.
• Use a lightweight ECAPA-TDNN-Small model (512 channels) with only 2–3 training iterations, achieving high performance despite simpler architecture and clustering.

Experimental results

Research questions

• RQ1 Can a self-supervised speaker verification system achieve state-of-the-art performance without relying on any human-annotated labels or large-scale supervised data?
• RQ2 How can the DINO self-distillation framework be adapted to improve speaker representation learning in the absence of negative pairs?
• RQ3 To what extent can cluster-aware training enhance model generalization by leveraging intra-speaker diversity across multiple utterances?
• RQ4 Can dynamic loss-gating and label correction significantly mitigate performance degradation caused by noisy or unreliable pseudo-labels in iterative self-supervised training?
• RQ5 Does multi-modal fusion of audio and visual features further close the performance gap between self-supervised and fully supervised speaker verification systems?

Key findings

• The proposed CA-DINO + DLG-LC framework achieves relative EER improvements of 22.17%, 27.94%, and 25.56% on Vox-O, Vox-E, and Vox-H test sets respectively, compared to the best-known self-supervised system.
• Even with only 2–3 training iterations and a smaller ECAPA-S (Small) model, the method outperforms prior state-of-the-art systems that used larger models and more iterations.
• The system achieves comparable performance to a fully supervised ECAPA-TDNN-Small model on VoxCeleb, despite using no ground-truth labels.
• Multi-modal integration of audio and visual features in the DLG-LC framework leads to a 1.191% EER on Vox-O, surpassing the fully supervised baseline of 1.010% when using adaptive s-norm for fair comparison.
• The dynamic loss-gate mechanism, based on GMM modeling of loss histograms, effectively identifies and filters unreliable pseudo-labeled samples, improving training stability and accuracy.
• Label correction via model posterior estimation prevents performance degradation from incorrect labels, outperforming simple data pruning strategies.