[Paper Review] VAE-based regularization for deep speaker embedding

Yang Zhang, Lantian Li+1 more|arXiv (Cornell University)|Apr 7, 2019

Speech Recognition and Synthesis22 references4 citations

TL;DR

This paper proposes a VAE-based regularization method to transform non-Gaussian x-vectors into more Gaussian latent codes, improving compatibility with PLDA scoring. By combining variational autoencoding with a speaker cohesive loss, the approach achieves state-of-the-art performance on SITW and CSLT-SITW datasets, outperforming both x-vectors and alternative regularization methods like PCA and AE.

ABSTRACT

Deep speaker embedding has achieved state-of-the-art performance in speaker recognition. A potential problem of these embedded vectors (called `x-vectors') are not Gaussian, causing performance degradation with the famous PLDA back-end scoring. In this paper, we propose a regularization approach based on Variational Auto-Encoder (VAE). This model transforms x-vectors to a latent space where mapped latent codes are more Gaussian, hence more suitable for PLDA scoring.

Motivation & Objective

• To address the issue that x-vectors from deep neural networks are unconstrained and non-Gaussian, degrading PLDA performance.
• To regularize x-vectors into a more Gaussian latent space using a Variational Auto-Encoder (VAE) architecture.
• To enhance regularization by introducing a speaker cohesive loss that constrains speaker conditionals.
• To demonstrate that VAE-based regularization and PLDA scoring are complementary, improving both cosine and PLDA scoring performance.

Proposed method

• A VAE is trained to map raw x-vectors to a latent space where the marginal distribution of speaker codes is approximately Gaussian.
• The VAE uses a reparameterization trick to enable end-to-end training of the variational posterior and prior.
• A speaker cohesive loss is introduced to encourage speaker conditionals to be more Gaussian, aligning with PLDA assumptions.
• The model is jointly optimized using reconstruction loss, KL divergence, and the cohesive loss to regularize both marginal and conditional distributions.
• The resulting latent codes (v-vectors and c-vectors) are used for downstream scoring with cosine similarity or PLDA.
• The method is evaluated on SITW and CSLT-SITW datasets, comparing against x-vectors, PCA, AE, and PLDA baselines.

Experimental results

Research questions

• RQ1 Can a VAE effectively regularize non-Gaussian x-vectors into a more Gaussian latent space suitable for PLDA scoring?
• RQ2 Does adding a speaker cohesive loss improve the regularization of speaker conditionals in the latent space?
• RQ3 How does VAE-based regularization compare to other unsupervised methods like PCA and AE in terms of performance on speaker verification tasks?
• RQ4 Is the VAE-based regularization complementary to PLDA, or does it replace the need for PLDA in scoring?
• RQ5 Does the VAE-regularized representation generalize better to out-of-domain settings like CSLT-SITW?

Key findings

• On the SITW Dev. Core set, the VAE-regularized c-vector achieved an EER of 3.31% with L-PLDA, outperforming the x-vector baseline (3.12% with L-PLDA) and cosine scoring (9.05% EER).
• On the SITW Eval. Core set, the c-vector achieved 3.53% EER with L-PLDA, significantly outperforming the x-vector baseline (3.80% EER with L-PLDA).
• In out-of-domain testing on CSLT-SITW, the c-vector achieved 11.98% EER with P-PLDA, showing larger gains than on SITW, indicating robustness to complex acoustic conditions.
• The v-vector and c-vector showed significantly lower skewness and kurtosis than x-vectors, confirming their more Gaussian distribution, especially at the speaker level.
• PCA and AE failed to produce suitable regularization, as evidenced by higher EERs and less Gaussian distributions, highlighting the importance of the probabilistic nature of VAEs.
• The VAE-based regularization implicitly regularized the prior distribution, making it more suitable for PLDA modeling, as shown by reduced kurtosis in speaker-level statistics.