[Paper Review] Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification

Zhengyang Chen, Sanyuan Chen+6 more|arXiv (Cornell University)|Oct 12, 2021

Speech Recognition and Synthesis9 citations

TL;DR

This paper proposes using weighted average pooling of self-supervised speech representations from all hidden layers of pre-trained models (Wav2Vec2.0, HuBERT, UniSpeech-SAT) to improve automatic speaker verification (ASV) with ECAPA-TDNN. The method achieves a 0.537% EER on VoxCeleb1-E, outperforming the VoxSRC2021 challenge winner on that trial, and demonstrates that lower layers of pre-trained models contain more discriminative speaker information than higher layers.

ABSTRACT

The speech representations learned from large-scale unlabeled data have shown better generalizability than those from supervised learning and thus attract a lot of interest to be applied for various downstream tasks. In this paper, we explore the limits of speech representations learned by different self-supervised objectives and datasets for automatic speaker verification (ASV), especially with a well-recognized SOTA ASV model, ECAPA-TDNN [1], as a downstream model. The representations from all hidden layers of the pre-trained model are firstly averaged with learnable weights and then fed into the ECAPA-TDNN as input features. The experimental results on Voxceleb dataset show that the weighted average representation is significantly superior to FBank, a conventional handcrafted feature for ASV. Our best single system achieves 0.537%, 0.569%, and 1.180% equal error rate (EER) on the three official trials of VoxCeleb1, separately. Accordingly, the ensemble system with three pre-trained models can further improve the EER to 0.479%, 0.536% and 1.023%. Among the three evaluation trials, our best system outperforms the winner system [2] of the VoxCeleb Speaker Recognition Challenge 2021 (VoxSRC2021) on the VoxCeleb1-E trial.

Motivation & Objective

• To investigate whether large-scale self-supervised speech representations can enhance state-of-the-art (SOTA) speaker verification systems like ECAPA-TDNN.
• To determine whether combining representations from all hidden layers of pre-trained models yields better performance than using only the final layer.
• To analyze the distribution of speaker-related information across different layers of pre-trained models.
• To evaluate the impact of model architecture, pre-training objective, and training dataset size on downstream ASV performance.
• To compare the effectiveness of self-supervised representations against conventional handcrafted features like FBank.

Proposed method

• The method uses a learnable weighted average of hidden-layer representations from pre-trained models (Wav2Vec2.0, HuBERT, UniSpeech-SAT) as input to ECAPA-TDNN.
• The weights are trained end-to-end during downstream training, allowing the model to learn which layers contribute most to speaker discrimination.
• The pre-trained models are fine-tuned jointly with the ECAPA-TDNN classifier on VoxCeleb1 and VoxCeleb2 datasets.
• The study compares models with different architectures (base vs. large), pre-training objectives (contrastive, masked prediction), and training data scales (XLSR, LibriSpeech, etc.).
• Ablation studies are conducted to evaluate the impact of freezing vs. fine-tuning the pre-trained encoder.
• Layer-wise importance is visualized via normalized attention weights to analyze where speaker information is most concentrated.

Experimental results

Research questions

• RQ1 Can self-supervised speech representations from large-scale unlabeled data improve the performance of a SOTA speaker verification system like ECAPA-TDNN?
• RQ2 Is it beneficial to use a weighted average of representations from all hidden layers rather than just the final layer?
• RQ3 Which pre-trained model architecture and pre-training objective yield the best performance for speaker verification?
• RQ4 Where in the pre-trained model (i.e., which layers) is the most discriminative speaker information located?
• RQ5 How does joint fine-tuning of the pre-trained model and downstream classifier affect system performance?

Key findings

• The proposed method with learnable weighted averaging of all hidden layers achieved a 0.537% equal error rate (EER) on the VoxCeleb1-E trial, outperforming the VoxSRC2021 challenge winner on that specific trial.
• The ensemble of three pre-trained models further reduced EER to 0.479% on VoxCeleb1-E, demonstrating the robustness of the approach.
• Self-supervised representations significantly outperformed conventional FBank features, achieving a relative EER improvement of approximately 30% on all VoxCeleb1 trials.
• Lower layers of the pre-trained models (especially in large-scale models like UniSpeech-SAT_Large and Wav2Vec2.0_Large) were found to contain more discriminative speaker information than higher layers, as indicated by higher learned weights.
• Fine-tuning the pre-trained model during downstream training led to greater performance gains on the larger VoxCeleb2_dev set than on VoxCeleb1_dev, suggesting data scale influences fine-tuning effectiveness.
• The Wav2Vec2.0_Large (XLSR) and UniSpeech-SAT_Large models performed best after fine-tuning, with the latter achieving the lowest EER of 0.537% on VoxCeleb1-E.