[Paper Review] Deep Speaker Vector Normalization with Maximum Gaussianality Training

Yunqi Cai, Lantian Li+2 more|arXiv (Cornell University)|Oct 30, 2020

Speech Recognition and Synthesis48 references4 citations

TL;DR

This paper proposes Maximum Gaussianality (MG) training for Deep Normalization Flow (DNF)-based speaker vector normalization, directly optimizing latent code Gaussianity instead of relying on maximum likelihood (ML) training. The MG approach significantly improves speaker recognition performance—especially with cosine scoring—by enforcing spherical between-class distributions and homogeneous within-class variances, outperforming ML-based DNF on both in-domain (SITW) and out-of-domain (CNCeleb) datasets, with notable gains even under low-data regimes.

ABSTRACT

Deep speaker embedding represents the state-of-the-art technique for speaker recognition. A key problem with this approach is that the resulting deep speaker vectors tend to be irregularly distributed. In previous research, we proposed a deep normalization approach based on a new discriminative normalization flow (DNF) model, by which the distributions of individual speakers are arguably transformed to homogeneous Gaussians. This normalization was demonstrated to be effective, but despite this remarkable success, we empirically found that the latent codes produced by the DNF model are generally neither homogeneous nor Gaussian, although the model has assumed so. In this paper, we argue that this problem is largely attributed to the maximum-likelihood (ML) training criterion of the DNF model, which aims to maximize the likelihood of the observations but not necessarily improve the Gaussianality of the latent codes. We therefore propose a new Maximum Gaussianality (MG) training approach that directly maximizes the Gaussianality of the latent codes. Our experiments on two data sets, SITW and CNCeleb, demonstrate that our new MG training approach can deliver much better performance than the previous ML training, and exhibits improved domain generalizability, particularly with regard to cosine scoring.

Motivation & Objective

• To address the limitation of maximum likelihood (ML) training in DNF models, which fails to ensure linear Gaussian latent codes despite model assumptions.
• To improve speaker vector normalization by directly maximizing the Gaussianity of latent codes, aligning with the ideal PLDA scoring assumptions.
• To enhance performance and generalizability, especially under low-data or out-of-domain conditions, using a more principled training criterion.
• To demonstrate that MG training enables better performance than ML-based DNF, particularly with cosine scoring, which is less sensitive to data mismatch.

Proposed method

• Proposes a new training criterion—Maximum Gaussianality (MG)—that directly maximizes the Gaussianity of latent codes produced by the DNF model.
• Replaces the standard maximum likelihood (ML) objective with a Gaussianity-optimization objective, using statistical measures such as Mardia’s multivariate kurtosis to quantify Gaussianity.
• Applies MG training to both between-class and within-class distributions, enforcing spherical mean distribution and homogeneous within-class variance.
• Uses invertible flow-based models (DNF) to transform speaker embeddings into normalized latent codes with improved distributional properties.
• Employs a two-stage training strategy: first normalize within-class distributions, then normalize between-class means, both under the MG criterion.
• Enables use of any invertible flow structure without requiring explicit entropy computation, simplifying model design and training.

Experimental results

Research questions

• RQ1 Can maximizing the Gaussianity of latent codes in DNF-based speaker normalization improve performance over maximum likelihood training?
• RQ2 Does MG training lead to better generalization on out-of-domain datasets, especially when using cosine scoring?
• RQ3 How does MG training affect performance under low-data regimes, such as few utterances per speaker?
• RQ4 To what extent does MG-based normalization enhance the effectiveness of cosine scoring compared to PLDA scoring?
• RQ5 Can MG training be applied effectively to both within-class and between-class distributions simultaneously?

Key findings

• MG training significantly improves speaker recognition performance on the SITW dataset, with a 31% relative reduction in equal error rate (EER) on in-domain tests compared to ML-based DNF.
• On out-of-domain testing with CNCeleb, MG-based DNF achieves a 9.3% relative EER reduction, demonstrating strong domain generalizability.
• With only 2 utterances per speaker, MG training maintains high performance, while ML-based DNF plateaus, showing superior robustness in low-data scenarios.
• Cosine scoring with MG-normalized vectors achieves performance comparable to or better than PLDA scoring, especially in out-of-domain settings, due to reduced sensitivity to data mismatch.
• Joint normalization of both between-class and within-class distributions via MG training yields the highest performance gains, confirming the importance of full distributional regularization.
• The MG criterion enables effective training of flow-based models without explicit entropy computation, broadening applicability to various invertible architectures.