[Paper Review] Large Margin Softmax Loss for Speaker Verification

Yi Liu, Liang He+1 more|ArXiv.org|Apr 6, 2019

Speech Recognition and SynthesisComputer Science32 references17 citations

TL;DR

This paper proposes an enhanced large margin softmax loss for speaker verification by integrating AM-Softmax with Ring loss and minimum hyperspherical energy (MHE) criterion, significantly improving discriminative feature learning. On VoxCeleb, the best system reduces EER by 15%, minDCF08 by 13%, and minDCF10 by 33% compared to baseline.

ABSTRACT

In neural network based speaker verification, speaker embedding is expected to be discriminative between speakers while the intra-speaker distance should remain small. A variety of loss functions have been proposed to achieve this goal. In this paper, we investigate the large margin softmax loss with different configurations in speaker verification. Ring loss and minimum hyperspherical energy criterion are introduced to further improve the performance. Results on VoxCeleb show that our best system outperforms the baseline approach by 15\% in EER, and by 13\%, 33\% in minDCF08 and minDCF10, respectively.

Motivation & Objective

• To improve speaker embedding discriminability and intra-speaker compactness in neural network-based speaker verification.
• To address the limitations of standard softmax loss, which does not explicitly encourage inter-class separation or intra-class compactness.
• To investigate the effectiveness of large margin softmax variants—ASoftmax, ArcSoftmax, and AMSoftmax—on speaker verification.
• To integrate auxiliary losses (Ring loss and MHE) to further enhance feature space separability and robustness.
• To achieve state-of-the-art performance on VoxCeleb with minimal backend complexity, ideally enabling simple cosine scoring without PLDA.

Proposed method

• Uses AM-Softmax with additive margin to enforce angular separation between speaker embeddings in the hypersphere.
• Applies Ring loss as a soft normalization to stabilize feature norm variance and reduce sensitivity to scale.
• Introduces minimum hyperspherical energy (MHE) loss to maximize inter-speaker distance variance on the hypersphere, improving overall separability.
• Employs feature normalization and angular margin in the output layer to focus on cosine similarity rather than magnitude.
• Trains the x-vector architecture with end-to-end optimization using the combined loss functions.
• Uses cosine scoring with LDA and PLDA for evaluation, but aims to eliminate PLDA dependency in future work.

Experimental results

Research questions

• RQ1 How do different large margin softmax variants (ASoftmax, ArcSoftmax, AMSoftmax) compare in speaker verification performance?
• RQ2 Can Ring loss effectively stabilize feature norm variance and improve training stability in speaker embedding learning?
• RQ3 Does the MHE criterion enhance inter-speaker separability by promoting uniform distribution of speaker centers on the hypersphere?
• RQ4 What is the optimal margin configuration for each large margin softmax variant in the context of speaker verification?
• RQ5 Can the combination of AMSoftmax with MHE loss achieve state-of-the-art performance while reducing reliance on complex backend classifiers like PLDA?

Key findings

• The best system, using AM-Softmax with MHE loss, reduces EER from 3.10% to 2.00%, a 15% relative improvement over the baseline.
• The AM-Softmax + MHE system reduces minDCF08 from 0.0169 to 0.0106, a 13% relative improvement, and minDCF10 from 0.4977 to 0.2487, a 33% relative improvement.
• AMSoftmax outperforms ASoftmax and ArcSoftmax across all metrics, with optimal margin $m_3 = 0.20$.
• The MHE loss reduces the variance of pairwise squared distances between normalized speaker weights, indicating more uniform and separable distribution on the hypersphere.
• Ring loss reduces feature norm variance, especially when combined with margin-based losses, but has less impact when AMSoftmax already constrains norms effectively.
• The AM-Softmax + MHE combination achieves the best performance, suggesting that maximizing inter-speaker separability through MHE is more effective than norm regularization alone.